1

Evaluating the Wider Outcomes of Schooling:

The Oasis ECHO Project

Technical Report

Dr Ruth Deakin Crick Professor Howard Green Steven Barr Adeela Shafi Dr Wen-Jung Peng

2

CONTENTS

Acknowledgements ......................................................................................................................................... 5

Introduction .................................................................................................................................................... 6

Background and Rationale ............................................................................................................................... 6

A Review of the Literature ............................................................................................................................. 10

Evaluating the outcomes of schooling .................................................................................................................. 11

Contextual Value-Added ....................................................................................................................................... 11

Multi-level modelling ............................................................................................................................................ 12

School effectiveness and school improvement (SESI) .......................................................................................... 14

The use of quantitative and qualitative data ........................................................................................................ 14

Wider outcomes of school .................................................................................................................................... 16

Effective leadership .............................................................................................................................................. 18

Moving forward .................................................................................................................................................... 19

Reframing Schools as Complex Living Systems .............................................................................................. 20

Understanding schools as complex systems ......................................................................................................... 22

Viewpoint One: Leaders leading learning and change in the community ............................................................ 23

Viewpoint Two: Teachers as learners ................................................................................................................... 24

Viewpoint Three: Students as learners - deep learning as a complex psycho-social system ............................... 26

The challenge of complex data collection and representation for schools ..................................................... 28

CHOICE OF APPROACH TO RESOLVING COMPLEXITY IN LEARNING COMMUNITIES ........................ 29

SYSTEMS THINKING FOR RESOLVING COMPLEXITY AND MANAGING UNCERTAINTY ...................... 30

RESEARCH DESIGN AND METHODOLOGY .............................................................................................. 31

SYSTEMS DESIGNING AS IMPROVEMENT RESEARCH DESIGN ................................................................ 31

CASE STUDY SAMPLE ....................................................................................................................................... 32

PHASING OF RESEARCH ................................................................................................................................... 32

STEP ONE: DESIGN MODEL AND MEASUREMENT PARAMETERS ........................................................... 33

STEP TWO: COLLECT AND ANALYSE ALL DATA .......................................................................................... 36

3

MAPPING THE DATA STRANDS ONTO THE CRITICAL SUCCESS INDICATORS ...................................... 40

STEP THREE: GENERATE AND ANALYSE PERIMETA MODEL .................................................................... 45

STEP FOUR: FEEDBACK TO LEADERS AND EXPLORE SOLUTIONS .......................................................... 50

STEP FIVE: EMBED NEW PRACTICES ............................................................................................................ 51

SUMMARY ............................................................................................................. Error! Bookmark not defined.

Summary ............................................................................................................................................................... 51

Data Analysis: Social science.......................................................................................................................... 52

Strand One - Development of students as learners in Year 7 ..................................................................................... 52

Strand Two - Student perceptions of the impact of the Education Charter ............................................................... 59

Strand Three - Stories of transformation .................................................................................................................... 70

Strand Four - Post-16 transition and progress to adulthood ...................................................................................... 72

Strand Five - Quality of relationships between parents/carers and the Academy ..................................................... 84

Strand Six - Learning of teachers: impact of CPD on the quality of classroom practice ............................................. 90

Strand Seven - Impact CPD on the quality of leadership and management and classroom practice ......................... 99

Strand Eight & Nine - Key Performance Indicators & Key outcome for Year 11 students – GCSE results ................ 108

Hiearchical process model .................................................................................................................................. 134

Strand One - Development of students as learners in Year 7 ................................................................................... 134

Strand Two - Student perceptions of the impact of the Education Charter ........................................................... 138

Strand Three - Stories of transformation .................................................................................................................. 142

Strand Four: Post-16 transition and progress to adulthood ..................................................................................... 143

Strand Five: Quality of relationships between parents/carers and the Academy .................................................... 146

Strand Six: Learning of teachers: impact of CPD on the quality of classroom practice ............................................ 149

Strand Seven: Impact of CPD on the quality of leadership and management and classroom practice .................... 155

Strand Eight and Nine: Context for student learning - Key Performance Indicators and GSCE results .................... 158

Triangulation: HPM and Social Science ............................................................................................................... 158

Discussion of Findings .................................................................................................................................. 159

ECHO Strand 1 - Development of students as learners in Year 7 ............................................................................. 159

ECHO Strand 2 - Student perceptions of the impact of the Education Charter ........................................................ 161

4

ECHO Strand 3 – Stories of transformation .............................................................................................................. 162

ECHO Strand 4 - Post-16 transition and progress to adulthood ............................................................................... 162

ECHO Strand 5 – Quality of relationships between parents/carers and the Academy............................................. 164

ECHO Strand 6 - Learning of teachers: impact of CPD on the quality of classroom practice ................................... 165

ECHO Strand 7 - Impact of Continued Professional Development (CPD) on the quality of leadership and

management and classroom practice ....................................................................................................................... 166

Usefulness of Perimeta data for leadership decisioning ......................................... Error! Bookmark not defined.

Practical and Technical Issues for Development ................................................................................................ 167

Conclusions and Next Steps ......................................................................................................................... 169

Bibliography and References ....................................................................................................................... 172

APPENDIX .......................................................................................................... Error! Bookmark not defined.

APPENDIX 1 GLOSSARY OF TERMS ............................................................................................................. 179

APPENDIX 2 OASIS CHARTER ....................................................................................................................... 179

APPENDIX 3 CRITICAL SUCCESS INDICATORS (CSI) ................................................................................ 181

APPENDIX 4 QUESTIONAIRES ...................................................................................................................... 182

STRAND ONE – STUDENTS SELF-REPORT QUESTIONNAIRE ON LEARNING POWER .............................. 182

STRAND TWO - STUDENTS PERCEPTIONS OF EDUCATION CHARTER ....................................................... 182

STRAND THREE – STUDENT NARRATIVE IN TERVIEWS ................................................................................. 184

STRAND FOUR – YEAR 11 LEAVERS PERCEPTIONS ........................................................................................ 185

STRAND FIVE - PERCEPTIONS OF PAREN TS AND CARERS OF THEIR CHILDREN'S ACADEMIES .......... 186

STRAND SIX - LEARNING OF TEACHERS ........................................................................................................... 188

STRAND SEVEN - SENIOR LEADERSHIP ............................................................................................................ 191

APPENDIX 5 FULL PERIMETA RESULTS ................................................................................................................ 195

APPENDIX 6 ......................................................................................................................................................... 211

5

ACKNOWLEDGEMENTS

The authors of this report would like to thank the following for their valuable contributions to the

ECHO Project:

The University of Bristol

Oasis Community Learning

Jerusalem Trust

Dr Helen Jelfs who undertook analysis of the teacher and student interview data

Claire Wilson who coordinated data gathering from the Academies at Oasis Community

Learning

John Fuller, Director of Education at OCL, for his support throughout the Project

The Principals and Staff of Oasis Academies, Enfield Academy, Lords Hill Academy and

Wintringham Academy for taking the time to be a part of this project. Academies have been

anonymised within this report.

6

INTRODUCTION

A continuing challenge for the education system is how to evaluate the wider outcomes of schools.

Academic results are important but other, less easily quantifiable measures of success make for a

complete education. For example, the development of students as life-long learners, employability

skills, citizenship, self-confidence, teamwork and emotional wellbeing are widely recognised as

essential qualities for individual success in adult life and for social cohesion. Unless methods are

found to evaluate these broader outcomes, the education system will continue to focus on a single

measure of school effectiveness: test/exam results. This report describes the rationale,

methodology and findings of a pilot research project that applied hierarchical process modelling to

schools as complex living systems, using software developed by engineers at the University of

Bristol, called Perimeta. The aim was to create a systems design which accounted for the full range

of outcomes valued by each school, collect evidence of success – in the form of quantitative,

qualitative and narrative data – and to model this using Perimeta software which returns visual

analytic feedback against each outcome in the form of the Italian flag. Red represents what is not

working, green represents what is successful and white represents what is not known, and is

therefore an area for organisational learning and development. The project involves three

Academies in the UK. The systems design which was developed to underpin this pilot study, is one

which recognises that the purpose of the school is to facilitate the learning and achievement of all

students and the core processes which are essential for fulfilling this are: leadership learning,

teacher learning and student learning.

BACKGROUND AND RATIONALE

Ten years ago Macbeath and McGlynn (Macbeath and Mcglynn, 2002) reviewed thirty years of

school effectiveness research and described the shift that had occurred from evaluating schools as

whole units to a more specific focus on what is happening in individual classrooms. They argued

that this shift should be complemented by a wider focus on school culture:

‘It is not a matter of either/or: school or classroom, management or teachers, teaching or learning. Measuring effectiveness means sharpening our thinking as to where we should give most attention and invest our energies at any given time and in the light of the priorities we pursue. And as we get better at it we recognise that in good schools the boundaries between different levels become so blurred that they defy even the most inventive of statistical techniques’. (2002:6)

They then go on to describe a model of evaluation that puts pupil learning at the centre, but set in

the context of a school culture that sustains staff learning, leadership that creates and maintains

the culture and an outward-facing dimension involving home and community. They continue:

7

‘In deciding what to evaluate there is an irresistible temptation to measure what is easiest

and most accessible to measurement. Measurement of pupil attainment is unambiguously

concrete and appealing because over a century and more we have honed the instruments for

assessing attainment (and used them) for monitoring and comparing teacher effectiveness’.

(2002:7)

School self-evaluation (SSE) involves schools evaluating their own performance in order to

propel improvement and, according to Ritchie (2007), to maintain a level of control on external

evaluations, within a target driven culture (particularly in the UK). School self-evaluation tends

to focus on leadership and its impact on school evaluation. When done rigorously it can provide

the tools to set objectives for school improvement so that all schools can be effective. The

distinction between assessment and evaluation is that the latter demands taking a step back

from a specific piece of work or programme of study and asking questions like: was the

experience worthwhile? What was learnt from the process? What might be done next time to

improve? How is the school developing effective learners? Assessment, on the other hand, is

more about measuring a more quantifiable and narrower set of school outcomes, most often in

the form of test or exam results (Sammons, 1999).

The early application of information technology as a tool to support data-gathering and analysis

for the evaluation of school performance, combined with a target-driven culture of school

improvement, has resulted in an ever more single-minded focus on quantifiable measures of

success in schools. Whilst the development of sophisticated statistical techniques such as multi-

level modeling (Goldstein, 1986) have attempted to address the complexity of the range of

variables operating at different levels within the system and the interactions between them, so

as to tease out a more nuanced account of what makes a school effective, there remains

however, a great reliance on assessment scores. Despite this recognition of the complexity of

evaluating school outcomes school evaluation has continued towards a reductionist focus on

test performance at the expense of a wider, more balanced range of outcomes. This movement

has been fuelled significantly by political imperatives. Davies (Davies, 2011) raises similar

concerns about the current restricted view of ‘success’ from the perspective of strategic

leadership. He suggests that, by focusing on too narrow a range of school performance

measures, the upward trajectory of pupil attainment might plateau:

‘Success can be seen in how children achieve academically, socially, spiritually, physically and emotionally; it is enabling children to be all they can be. The difficult question is how do you know that you have been successful? Standardized test scores, even when adjusted for value-added dimensions, tell only part of the story. Two challenges emerge. One is that measuring success by easily quantifiable measures is to ignore that some aspects of success are recognised by indicators which point to success, but do not by any means fully explain or measure that success. Secondly, approaches that make schools successful initially may not be the ones that are necessary to take them on to higher levels of performance, so that isolating what approaches lead to sustainable success is difficult. A good example of this is the difference between shallow and deep learning. Coaching children for standard

8

assessment tasks (SATS) tests may increase short term results and the school would be considered successful. However, instead of putting in enormous efforts every year to boost results, a longer-term and more sustainable approach would be to involve ‘deep learning’ approaches that develop a learning culture in individuals and the school’. (2006:12)

However as technology has continued to develop there are new opportunities emerging for the re-

presentation of complex data and the development of learning analytics which offer new ways of

responding to complexity in learning communities (Buckingham Shum and Deakin Crick, 2012).

Combined with insights into systems thinking, systems design and systems modelling developed in

the corporate sector and engineering in particular, new technologies offer a potentially richer

approach to evaluating the wider purposes of education, taking a broader view of both processes

and outcomes which is a better representation of the ‘messy reality’ of life in learning

organisations.

The ultimate aim of this project is to develop useful ways of assessing complex processes in

learning communities which encourage powerful learning and feedback at all levels of the learning

system. What we are seeking for is a richer approach to evidence based self-evaluation which will

enable a holistic approach to learning and performance, a participatory culture of high aspirations

and deep learning and a context of continual improvement where standards of pupil attainment

continue to improve. School self-evaluation can provide the tools to set objectives for school

improvement so that all schools can be effective. Leadership has been shown to be a significant

component of school improvement and research has been continuing to support this notion

through the decades (see Hallinger & Heck, 2010). It reinforces the growing support over how

effective leadership supporting teaching and learning and promoting professional development and

change can lead to improving school outcomes.

However, this is a particular challenge for schools and groups of schools which have a broader view

of education and a desire to extend the measures of school performance beyond the easily

quantifiable. These include schools underpinned by alternative philosophies such as Co-operative

schools or Humanscale schools as well as many faith based schools. Writing from the perspective of

Church schools, for example, (Grace, 2002) refers to this challenge when he writes:

‘Such religious and transcendent purposes, a sense of mission rather than simply of provision, gives…‘depth to a schooling process that is otherwise dominated by a rhetoric of test scores, performance standards and professional accountability’.

Bryk et al (Bryk et al., 1993) also point to the need for more studies of the inspirational ideology

which animates many Catholic schools. They admit the scepticism which this idea produces,

especially in a research culture strongly influenced by secular and positivistic assumptions, but

make a powerful case for such research:

‘Some may question our claim of a causal role for this inspirational ideology ….unlike the effects

of academic organization or school structure, which can be largely captured in regression

9

analysis and effect sizes, estimating the influence of ideology is a more complex and less certain

endeavour. Ironically, these effects are harder to study and yet also more pervasive … To ignore

the importance of ideology because it cannot be easily captured in statistical analysis or

summarised with numbers would be a serious mistake. Statistical analysis can help us to see

some things but they can also blind us to the influence of factors that are beyond their current

horizons’. (1993, 303-4)

This pilot research project takes on the challenge of evaluating the wider intentions and outcomes

of schools more systematically, drawing on quantitative, qualitative and narrative data. Good

schools have always sought the views of students and parents in taking a broader view about how

well the school is doing and planning for change. In the UK, OFSTED inspections continue to

monitor broader aspects of education like spiritual, moral, social and cultural development but this

evidence typically becomes secondary, rather than integral, when overall and final judgements are

made about performance. The fundamental questions behind this project are these: if we

acknowledge the importance of student attainment but also have the ambition to educate our

students for a set of broader outcomes, how can we know how well we are doing and what we

might need to do to improve? How can we do this in systematic, sustainable and convincing ways?

10

A REVIEW OF THE LITERATURE

Evaluating outcomes of schools, known as school effectiveness research (SER), originates from the

phenomenon of the ineffective school (Edmonds, 1979). It has been a specific field of study only in

the last 40 years, but in that time has been emerging with a growing international profile. With

education high on the agenda for most governments, evaluating the outcomes of schools has been

the subject of political debate not least for those countries where education is state-funded, with a

focus on improving school standards and political accountability (Teddlie and Reynolds 2000).

Although discussions around schooling and its effectiveness have raged for some time, research

began to be more focussed in the 1970s, particularly in the UK and US with the works of Coleman

(1966), Jencks et al (1972) and Rutter et al (1979). These studies argued that whilst schooling

generally makes a difference, the particular school attended has little impact on outcomes. These

early studies reinforced the sociological determinism argument that factors such as IQ, gender and

social background are more important for school outcomes than what actually happens in school.

They further argued that whilst some schools do seem to do better than others, it was more due to

their student intake than the effectiveness of the school. Unease and at times hostility (eg

Reynolds 1997) at claims made by these studies led to the development of the now established

field of school effectiveness research (also recently called educational effectiveness research EER to

encompass wider education services such as pre-school, further and higher education), with a

dedicated journal School Effectiveness and School Improvement established in 1990.

Different schools had differing levels of effectiveness and in order to address the issues brought to

the fore by the earlier studies, it was important to disentangle the complexity of factors which

contribute to school outcomes. These factors include ability, prior attainment, personal

dispositions, age, gender and family background amongst others, which combined with educational

experiences in school impact on later attainment progress and development (Sammons, 1999). The

multitude of factors and their interactions have meant that school effectiveness research has had

its fair share of issues, both on philosophical and methodological grounds. Some of these will be

explored in the next sections.

School effectiveness research is not to be confused with school improvement (SIR) research which

presents itself as a separate and distinct domain, specifically exploring how schools can be

improved. How schools improve is a ‘process and not an event’ (Reynolds et al., 1996 p.14),

therefore, cannot be measured in the same ways as school effectiveness. It is because school

effectiveness research and school improvement research are conceptually different in several ways

(Reynolds et al., 1996) that they have maintained a parallel existence as domains of enquiry despite

being essentially components of the same thing: how to make schools better. However, if they are

two sides of the same coin then there must be more connecting the two fields of research. In

recognition of this there is growing trend of bringing both these fields together and this will be

explored later in this literature review. But first it would be useful to examine the issues in school

11

effectiveness research, as it is this which has traditionally been most interested in evaluating the

outcomes of schooling.

EVALUATING THE OUTCOMES OF SCHOOLING

Whilst the research in evaluating outcomes of schools has grown in stature, it has been subject to

much debate on philosophical and methodological grounds. The philosophical and political debate

is focussed around (i) policy issues, where raising standards and increased accountability became

the main discourse and (ii) the philosophical debate which centred around epistemological issues:

values in education, the purpose of schooling, what is a ‘good school and so on. These are justified

and on-going debates (White & Barber 1997) and influence a third domain of enquiry: (iii) the

methodological debate. It is the methodological debate which is the main focus of this review.

Mainly led by statisticians, the debate surrounding methodology in school effectiveness research

centred on the analyses of data, stemming largely from the controversial Rutter et al (1979) 15000

Hours study. This study was influential, primarily because of the claim that pupil attainment was

due to social background and that school made little difference to life chances. However, the study

was severely criticised for its small sample size and, importantly, because it did not fully take

account of the complex nature of schools and the various impacts on outcomes in its statistical

analysis (Heath & Clifford, 1980; Sammons 1999). This sparked a growth in sophisticated statistical

analyses of school effectiveness data, such as multi-level modelling, comparing various levels of

school and student data, both between and within schools. It has become an essential and integral

part of the research on school effectiveness (Goldstein 1987).

CONTEXTUAL VALUE-ADDED

School effectiveness can be defined as a school where students progress more than might be

expected, given the school's student intake. In that sense, an ineffective school is where students

make less than expected progress, given its intake (Mortimore 1991a). This definition emphasises

the importance of ‘intake’ and refers to the now familiar term ‘value-added’.

It is fairly well established that raw-score indicators of pupil attainment are not a reflection of the

school effect, but are more an indicator of the school intake (Gorard, 2010). It has been

acknowledged by governments and policy makers that to compare schools based on these raw

scores is unfair and an inaccurate reflection of a school’s effectiveness. Consequently contextual

value-added (CVA) is a regular feature in the discussions of school effectiveness for researchers,

governments, policy makers and to a certain degree parents. The CVA scores take account of the

different starting points of a pupil and include indicators such as prior attainment, socioeconomic

status (SES) which is usually assessed by eligibility of students for free school meals (FSM), school

size and pupil mobility amongst others. School effectiveness research will include such data or

variations of it in their analysis of school effects and as claimed by the DCSF (2007) provided a much

fairer and more valid way to compare school effectiveness.

12

Contextual value added (CVA) is calculated using a complex statistical analysis (see Gorard 2010 for

a more detailed description) and has become the accepted way in which to ensure schools are

compared on a more like-for-like basis - indeed it is infinitely better than using raw scores as

indicators. However Gorard (2010) has argued in his recent paper that it is time to move on from

school effectiveness research which relies solely on CVA, primarily on the basis that CVA

calculations assume ‘perfect datasets’. In reality, data is often missing or data from different

datasets cannot be matched appropriately. For example, data from Pupil Level Annual School

Census (PLASC) and National Pupil Database (NPD) do not always map neatly on to one another.

Whilst, data analysts can claim that these are random and that they can statistically account for or

‘clean’ such data, it does not remove the issue that the data is not complete or of good quality. If

data at this early stage is incomplete or inaccurate, then the value of the CVA needs to be

questioned (Rowe, 2009).

Furthermore, Gorard (2010) advocated that such data limits what education is all about. He argues

for a refocus in enquiry into school effectiveness onto school processes, fostering outcomes other

than the tradition attainment score. These are often referred to as the non-cognitive outcomes of

schools, referring to pupil development in the area of social justice, aspirations and citizenship – all

of which are largely ignored in school effectiveness research. This could be because such a focus

would rely more on qualitative data which might be difficult for the quantitative researcher to

genuinely embrace. The complexity and vastness of the factors which affect school outcomes has

pointed towards the need to collect data in the form of ‘facts and figures’ from a wide range of

schools which has driven the movement towards large scale quantitative data collection.

Additionally, different value-added models themselves can yield different results. Thomas &

Mortimore's (1996) study compared different multi-level models and argued that if the data is rich

and wide-ranging then the school-context is not significant in predicting pupil outcomes. But

otherwise, even CVA seems to explain pupil intake only if the data is not of a good quality,

reinforcing Gorard’s (2010) point about the quality of data. Their analysis also emphasised school

level effects in terms of the effectiveness of different departments, especially for pupils with

differing levels of prior attainment. This reflected the growing plethora of criticisms of SER on the

use of aggregated data at the school level because an aggregate comparison of schools risks the

under estimation and potential misrepresentation of school effects (eg Willms & Raudenbush 1989;

Goldstein 1986). The development of multi-level modelling has provided a tool to overcome some

of the criticisms levelled at the methodology in SER.

MULTI-LEVEL MODELLING

The traditional statistical analysis used in school effectiveness research was OLS regression and

whilst it enabled sufficient analysis of data, it was criticised that it did not truly enable analysis of

the different levels of effect on school outcomes, producing over or under-estimates of what was

really reflected in a dataset. The development of multi-level modelling in the 90s as a statistical

13

package available commercially allowed data to be analysed at several levels (Fitz-Gibbon 1996).

Multi-level modelling enabled analysis to be conducted at sector, district, school and student level

across different time points with up to 15 levels available. Many considered it to be revolutionary

in the way it has provided a solution to many of the methodological criticisms of school

effectiveness research (Teddlie, Reynolds & Sammon 2000). Described as the answer to the issue

of units of analysis in school effectiveness research and that of comparing data across different

time points, it appeared to be the solution to many methodological problems. It also meant that

longitudinal studies could be assessed appropriately using this multi-level technique, resulting in a

growth in longitudinal studies (eg Mortimore et al, 1988) which addressed another criticism of

school effectiveness research (Teddlie and Reynolds, 2000). Furthermore, there has been a

growing tendency to focus, not just on schools as a unit of analysis, but on individual classrooms

(MacBeath and Glynn, 2002) and the role leadership plays in influencing what happens in a

classroom (Sammons, 2011), demonstrating how school effectiveness research has become more

sophisticated and detailed.

Whilst multilevel modelling was considered ground-breaking in terms of school effectiveness

research, more recently the ‘honeymoon period’ has been wearing off. Revisiting many of the pre-

multilevel analysis data, several studies have revealed similar results, whichever the mode of

analysis, especially at the school level (eg Fitz-Gibbon 1991a, 1995a, 1996; Gray et al 1995),

particularly if the size of the unit of analysis got close to 30. This seems to suggest that although

multilevel data has provided greater opportunities for analysis, more traditional methods such OLS

regression analysis (admittedly a more simpler analysis) are still useful. Consequently, Teddlie &

Reynolds (2000) argue that older studies which were disregarded because they used simpler

analyses should be reconsidered so that a decade or so of research is not lost.

The development of sophisticated methods of data analysis, such as multilevel modelling, structural

equation modelling, growth curve modelling, regression-continuity modelling, mean there is still

great emphasis on quantitative data collection analysis. There has been concern about the

excessive interest in the statistical analysis of data (Scheerens, 1992) with comparatively very little

discussion of the use of qualitative data and its analysis which reflects a general methods debate in

educational research (eg Lincoln and Guba, 1985; Tashakkori and Teddlie, 1988, 2003; Patton,

1990). This debate has fuelled and advocated the use of more diverse research methods in school

effectiveness research.

Consequently, school effectiveness research (SER) is often criticised for measuring a very narrow

set of school outcomes: namely attainment test scores, adjusted for the contextual value added

(Sammons, 1999). But as already discussed, these are fraught with issues. A further issue in SER is

that schools are complex communities in their own right where the student experience is far more

diverse than what can be reflected in the measurement of attainment scores (Levin, 2012). In that

sense, attainment in the form of test scores cannot be the whole picture of school effectiveness.

(Teddlie & Reynolds, 2000; Sammons, 1999). What needs to happen is for a deeper exploration of

14

what happens in schools in terms of the non-cognitive outcomes of schools, such as citizenship,

social cognition and well-being in addition to academic outcomes (Van de Wal, 2007). These

outcomes are often best explored using qualitative data alongside the quantitative. However,

these issues have usually been the domain of school improvement research (SI) rather than SER,

serving to explain the apparent professional polarisation of the two genres of research (Gray et al

1995).

SCHOOL EFFECTIVENESS AND SCHOOL IMPROVEMENT (SESI)

The rise of school effectiveness research (SER) stimulated the growth of school improvement (SI)

research but attempts to combine them have not always been successful (Creemers, Kyriakides and

Sammons 2010). SI primarily uses qualitative data often in the form of case histories and narratives

with an emphasis on practical knowledge and engagement with practitioners. Chapman (2011)

emphasised that the SI community advocate that improvement comes from within an organisation

not beyond it.

Until fairly recently, the field of school effectiveness research (SER) and school improvement have

generally been treated as distinct and separate with their own histories and traditions. This is

somewhat surprising because school effectiveness research can inform school improvement

practices which can then propel further research based on what is revealed at school improvement

level (Day et al, 2008). Thus, they do appear to be two sides of the same coin, although tensions

between the two have meant that they have often been treated distinctly where SER tends to have

its focus on exploring differences between schools and their outcomes, adopting a primarily

positivist stance (Sammons, 1999). The main form of data collection is quantitative and the results

are more about a picture of ‘what is’ which can be compared to other similar schools. In other

words, SER presents a picture of the effective or ineffective school and what characterises it as

such. School improvement research, on the other hand, is more focussed on a school and its

process of improvement (Chapman et al., 2011). The research tends to use more qualitative forms

of data with a concentration on teachers, school processes and the journey of school improvement

rather than the outcomes. In short, SI research is more focussed on making schools a ‘better place’

(Reynolds et al, 1996). Because of their different epistemologies and methodologies, they have

tended to remain distinct and have perhaps become a victim of academic ‘paradigm wars’ (Gage,

1989). However, as mixed methods research is becoming more widely used and described as the

third methodological community (Johnson et al 2007), minimising the weaknesses of each method

alone, educational researchers are increasing using mixed methods research designs, attempting to

combine school effectiveness and school improvement research (Teddlie & Sammons, 2010).

THE USE OF QUANTITATIVE AND QUALITATIVE DATA

As school effectiveness research seeks to identify and measure school effectiveness, the vast

majority of its studies have focussed on large scale longitudinal or cross-sectional data from a range

of schools in order to draw conclusions. However, as discussed, it is becoming more appropriate to

15

consider schools as organisations which are continually changing and evolving and are thus dynamic

places (Mulford, 2013) in which constant change is the norm. If this is the case, then taking a

‘snapshot’ of a school as is done in SER in order to judge effectiveness, is at best limited and at

worst simplistic. However it also represents a more serious failing to address issues of equity by

valorising a narrow view of educational achievement (Chapman and Gunter, 2009).

A study by Day, Sammons & Gu (2008) combined qualitative and quantitative methodologies to

explore teacher lives’ and their impact on teacher effectiveness in school. They claim to go beyond

simple integration of qualitative and quantitative methods towards synergy, pointing out that

simple integration is not enough and that the findings from the qualitative and quantitative strands

of research need to be synergised, creating deeper understanding and meaning which would not be

possible by simple integration or the sum of the two strands. Through their study they explored the

impact of early/mid-career teacher and end of career teacher commitments, using both

quantitative and qualitative data, initially by integrating the two types of data throughout where

one informed the other. Their conclusions about life phase and relative effectiveness were more

informed through the synergy of the quantitative and qualitative elements. They point to using

mixed methodologies in a synergistic as well as integrative way to truly benefit from the two

methods and create greater and deeper understandings. However, their discussion did

acknowledge the difficulties, particularly because often the qualitative and quantitative aspects of a

study are separate until the integration stages. This can be due to practical difficulties, for example

collecting quantitative data and then trying to integrate it with the corresponding qualitative data

may mean there is often a time lag, preventing truly concurrent dialogue. Nevertheless, the

engagement with the data leading to new synergies was greater than the individual contributions

of qualitative or quantitative data and therefore, important in understanding teacher effectiveness

based on career phase. This study is an example of how school effectiveness research is making

tentative moves towards mixed methods.

However, whilst this paints a picture of a growing sense of harmony between quantitative and

qualitative methods, it would be naïve to assume that both methods are being given equal

weighting when it comes to comes to reporting and drawing conclusions (Creswell, Shope, Plano

Clark, & Green, 2006). This may be because traditionally researchers fall into either the

quantitative or qualitative camp, usually with differing ontological, epistemological and micro-

political commitments, resulting in the polarisation of these methods, or the incompatibility thesis

(Guba & Lincoln 1994). Because many school effectiveness researchers come from a quantitative

tradition and are attempting to include qualitative data, the end analysis often still ends up skewed

towards the quantitative (Creswell et al, 2006).

This could also be due to the perception of uncertainty in the appropriate analysis of qualitative

data. Quantitative data collection has its origins in the positivist tradition involving the collection of

numerical data (Reynolds, Bollen, Creemers, Hopkins, Stoll & Lagerweij, 1996), whereas qualitative

data is laden with meanings, interpretations, explanations of language, behaviour and culture

16

which are open to inconsistencies, uncertainties and alternative perceptions. This of course is a part

of the lure of an interpretivist epistemology, serving as a source of deeper understanding of

complex social phenomena (Mathison, 1988). Thus, in this way qualitative data often get

‘relegated’ to ‘soft data’ (Creswell et al., 2006) by a primarily quantitative researcher. Because of

this and the school effectivenss research community’s focus on identifying effective schools,

qualitative data has become the domain of the school improvement field. Whilst facts and figures

are also open to interpretation (Scott 2000), many researchers believed that relevant stakeholders

would only take notice of quantitative research (Reynolds et al 1996). For example, exam grades

are a statement of what a student has achieved in a particular school at a particular point in time.

This grade is assessed irrespective of the process or whether the student had a good day, whether

the teacher was effective and all the other issues that may have influenced that exam grade. By

not considering the processes which led to this grade, interpretation is limited and offers little in

the way of how a school might improve this exam grade. These factors could be quantified, for

example, in the number of revision sessions or the number of hours a school dedicated to

delivering the subject. But school effectiveness research tends not to explore the processes

involved in improving grades. In this way, school effectiveness research needs to move beyond

what makes an effective school to concurrently exploring what processes within a school make it

effective in order to drive school improvement practices (Stoll & Fink 1992). In other words, it is

time for school effectiveness research and school improvement research to be part of the same

story. But in order for this to be the case, there needs to be greater acceptance of the methods

traditionally used by each of the research communities (Morgan, 2007).

This should not be impossible as adopting mixed methods in SER and SI research is compatible with

multilevel modelling which focusses in on different levels, such as the school, classroom, teacher,

student and which are inter-connected and/or nested within one another. Qualitative methods can

be employed to explore these inter-connected relationships in a deeper way, allowing more diverse

research questions to be addressed and which the quantitative method alone stops short of

(Reynolds et al 1996). In order to explore the complexity of SER and the range of levels which

multilevel modelling aimed to address, it would be appropriate to use qualitative data but which

would have to be fully integrated into the research design creating synergy in the findings which

would be greater than the sum of the qualitative and quantitative data analysis. It would also

bridge the gap between SER and SI making them more integral and reciprocal. The result could be

more effective and more improving schools (Mortimore 1991).

WIDER OUTCOMES OF SCHOOL

NON COGNITIVE OUTCOMES

Wider outcomes of schooling have become more interesting to researchers as the agenda for

education (in the UK at least) now includes, for example, citizenship as part of the National

Curriculum since 2002. Personal, social, health and economic education (PSHE) is also a non-

17

statutory subject, but which many schools choose to incorporate because the Office for Standards

in Education (OFSTED) assesses schools on this as one of their criteria. These are not academic

outcomes, but ones which schools and other educational establishments are expected to

incorporate into their education programmes. The 2010 Education White Paper emphasised the

importance of PSHE in helping young people develop the skills needed to make important decisions

in life including a healthy lifestyle and financial choices. Non-cognitive outcomes, such as

citizenships, social cognitions, well-being are also seen as important element of education (eg

Reynolds, 1992 ; (Isac, Maslowski, Creemers, & van der Werf, 2013) and as advocated by Teddlie

and Reynolds (2000), school effectiveness research needs to be multidimensional incorporating

both cognitive and non-cognitive outcomes (Opdenakker & Van Damme, 2000), especially as claims

are being made that effective social skills are more likely to determine life chances in the 21st

century (Mulford, 2013; Holmlund & Silva, 2009) This reinforces the call for using quantitative and

qualitative data in order to explore these phenomena more deeply.

However, wider outcomes of school have not featured in school effectiveness research for several

reasons. Firstly, they are not easily quantified or measured (Knuver & Brandsma, 1993), secondly,

because politically school systems have focused on academic attainment rather than the personal

and social development of the individual and so questions as to whether schools should be held

accountable for these are often raised (Knuver and Brandsma (1993). Thirdly, because such

research would require qualitative data which is not always considered easy to collect or analyse

and stakeholders do not always consider the findings seriously (Reynolds, 1996). Nevertheless,

recent research into non-cognitive outcomes has been growing, although has often been in relation

to how they may or may not be correlated with cognitive outcomes i.e. academic attainment,

rather than as outcomes in themselves. For example, the Van Landeghem, Van Damme,

Opdenakker, De Frairie, & Onghena (2002) explored the effect of school and class on non-cognitive

outcomes and reiterated the findings of other studies, that an effective school does not necessarily

equate to a school with effective non-cognitive outcomes (Opdenakker & Van Damme, 2000). They

used multilevel modelling on a range of survey data, but Van der Wal & Waslander (2007) discuss

how neither these nor their own study were able to draw firm conclusions on the trade-off when

focussing on either cognitive of non-cognitive outcomes. This reflects the notion that the

relationship between cognitive and non-cognitive outcomes is yet to be fully established. Van der

Wal & Waslander's (2007) research also used survey data and multivariate multilevel analysis. It is

noteworthy that few studies on school effectiveness research, even those which focus on non-

cognitive outcomes, make use of more qualitative data.

Furthermore, if at the school level, data is collected via interviews or case stories (qualitative), it is

not always used for setting objectives for improving outcomes. Rather the data tend to be used to

inform school improvement processes (Harris, 2001). This does not necessarily translate into a basis

from which to set objectives which are still largely determined by school effectiveness studies. This

once again points towards the need to integrate school effectiveness and school improvement

research so that their relative strengths are combined, and both contribute to setting objectives for

18

school improvement. Armstrong et al (2012) argue that the field of school effectiveness and

improvement research requires a dramatic change and should be renamed educational

effectiveness and improvement (EEI) encompassing all educational establishments, so that

effectiveness and improvement go hand in hand.

EFFECTIVE LEADERSHIP

A further development in school effectiveness and school improvement studies is a focus on school

leadership stimulated by the study of academically improved and effective schools. It has been

shown that effective and improved schools have a clear vision of their goals and which actions to

take to reach them (Hodges 2000). Leadership studies have been moving towards the centre of

school effectiveness and school improvement research (Teddlie and Reynolds 2000). A mixed

methods study by Sammons (2011) (which was part of a larger 3 year project commissioned by the

DCSF and summarised by Day et al (2010), found that leadership effects influence student

outcomes by their influence on teachers and on the creation of a favourable school climate. The

wider project, of which this paper was a part, reported on 20 case studies of qualitative data which

were thematically analysed and resulted in a matrix of the main themes from across the studies.

This was a study which made an attempt to integrate and create a synergy of the qualitative and

quantitative strands. It is worth noting that reports or research commissioned by policymakers is

more likely to be mixed methodologies simply because there is likely less commitment to particular

ontological positions than in academic circles. Nevertheless, in the case of this research, the lead

researchers are academics which demonstrate a genuine move towards mixed methods research.

The focus on leadership and its impact on schools is associated with school self-evaluation (SSE)

which involves schools evaluating their own performance in order to propel improvement and

maintain a level of control of evaluation, within an externally imposed target driven culture

(particularly in the UK) (Ritchie, 2007). When done rigorously SSE can provide the tools to set

objectives for school improvement so that all schools can become more effective. Leadership has

been shown to be a significant component of school improvement and research has been

continuing to support this notion through the decades (see Hallinger & Heck, 2010). It reinforces

the growing support for effective leadership focusing on teaching and learning and promoting

professional development as a process driver for school effectiveness and improvement.

Hallinger and Heck (2010), however, also point out that many leadership studies rely on case

studies and cross-sectional surveys. They rightfully advocate the need for longitudinal studies in

order to document process and change. But whilst this is important, they fail to recognise the value

of the case study data which have informed leadership studies and have enabled the current

conclusions of leadership to be made. Nevertheless school self evaluation has also been shown to

be an effective way in which school leadership can be been improved. This is despite SSE often

being criticised for lacking validity and reliability (see Kyriakides & Campbell, 2004 for a detailed

19

critical analysis), emphasising that external evaluation has to be a part of an overall approach to

school self-evaluation (Kyriakides & Campbell, 2004).

Alternatively, other studies of leadership have continued to rely on quantitative data to study its

effects on classroom practice and ultimately student outcomes. For example, Leithwood & Jantzi

(2006) used survey data from 2,290 teachers from 655 primary schools to analyse transformational

leadership against student achievement as measured by SATs scored at the end of Key Stage 2.

They found that leadership did affect teachers’ classroom practice but not student achievement.

But this could be because student outcome was measured by the narrow focus of academic

attainment (SATs) scores which as discussed cannot present a complete picture of what are

essentially qualitative qualities of a school.

This further advocates the use of more qualitative and perhaps longitudinal mixed method studies

(Kyriakides & Campbell, 2004) which could help produce a model of lasting impact on school

improvement helping to set targets and objectives as a whole school approach.

MOVING FORWARD

This literature review has aimed to untangle the research on the outcomes of schools as there are

various layers which overlap and are mutually re-inforcing. These moved from school effectiveness

research which explores differences between schools (largely using quantitative data) to school

improvement research (which uses primarily qualitative data) to a focus on the study of leadership

and its impact on school improvement to the more recent study of school self-evaluation, which is

a still developing field. What appears evident is that research on how schools operate has

proceeded on parallel but often separate tracks: effectiveness, improvement, leadership or self-

evaluation. However, schools are learning communities and organisations with complex structures

and processes and while it is tempting to break down these components and study each part

individually, it is important to bring the parts back into a whole in order to fully understand

schooling and its wider outcomes in the fullest sense.

One of the reasons for the distinctions between the fields lies in the methodologies available and

throughout the studies in this review there has been an extensive reliance on either quantitative or

qualitative data. Their ontological and epistemological foundations seem to have encouraged this

divide to continue. This is despite the regular calls for employing more mixed methods designs

(Charles Teddlie & Reynolds, 2000) across the genres. Influential researchers such as Muijs (2006,

2012) do make efforts to point out the ‘risk’ of, for example, attributional bias, in qualitative data,

which refers to an interviewee attributing success to themselves or internal factors and setbacks

down to external factors to do with the organisation, policy or management. However, this

emphasises further that it is important that to move on from the polarisation of these

methodologies to work towards developing more sophisticated analysis tools for qualitative and

narrative data as well as for quantitative data. It is not sufficient to brush off qualitative data on

the pretext that it is difficult to analyse or that it cannot be ‘trusted’, rather it is an opportunity to

20

explore qualitative and narrative data analysis as a field of research in itself. This is the point at

which qualitative and mixed methods designs will truly flourish, where qualitative data will

genuinely inform how schools are evaluated in a wider sense, informing how schools set their

objectives for improvement.

Developments in computer packages for analysing qualitative data (QDA), such as NVivo or

MAXQDA have been extremely useful in helping manage large amounts of data and taking on the

burden of some of the labour intensive aspect. However, computer-assisted qualitative data

analysis software (CAQDAS) cannot ‘analyse’ the data or fully explore the intricacies of qualitative

data (Bazely, 2007), which is primarily a human activity because it involves meaning making and

interpretation. Others criticisms, particularly made by positivists, have been levelled at the

proximity the researcher has to qualitative data. This could be minimised by the use of CAQDAS

with their extensive search, access and coding facilities potentially reducing a reliance on ‘first

impressions’ when reading qualitative data (Garcia-Horta and Guerra-Ramos, 2007) and make for

more robust interpretations (Bergin, 2011). This makes the process of the analysis of qualitative

data more open to scrutiny and thereby greater transparency – addressing another criticism

levelled at qualitative data analysis (Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005).

However these packages are still being developed and are comparatively some way from the

sophisticated statistical packages designed for quantitative data. They, therefore, deserve further

exploration and development.

Evaluating wider outcomes of schooling is a complex challenge because of the complexity of factors

which are inter-connected and inter-related, and the differing 'truth claims' of different genres of

research. This means that it is essential to develop research designs and methodologies which truly

explore all facets of this complicated process of schooling.

REFRAMING SCHOOLS AS COMPLEX LIVING SYSTEMS

One of the big challenges emerging from this literature review is how to harness the synergy that

might be created through the inter-action of different approaches to knowledge generation and

use and their differing approaches to truth claims. Such synergy might better inform leadership

decisioning in schooling than a focus on only one method at the expense of others, because it

would reflect the complex reality of schools. However, not only does this challenge include the

knowledge generated by those studying schools (i.e. researchers) but, significantly, it includes the

experiential professional knowledge generated by leaders and even the connected intelligence of

the whole community, including students. This is surely a complex challenge which is impelling in

21C conditions of risk, uncertainty and diversity, and the ubiquitous use of technology and data

offers new opportunities to engage with it.

21

In order to address the main aims of this project it is thus important to consider schools in all their

complexity. Such complexity is a fundamental issue for education and schooling and, as has been

demonstrated in the review of the literature, is represents a challenge in educational research.

Schools have multiple stakeholders and multiple outcomes. A school’s core processes of student

learning and achievement are themselves complex and dynamic and cannot be reduced to, or

described by, a single variable. As Wheatly and McCombs (Wheatley, 1999, Wheatley and Kellner-

Rogers, 1998); (Mccombs and Whisler, 1997) argue, a school is a living system and in order to

understand improvement and change in schooling we need to take this into account. A complex

living system is one which is self-organising, purposeful, layered, interdependent and operating 'far

from equilibrium' (Davis and Sumara, 2006); (Checkland and Scholes, 1999). This means that there is

no single blue print for improvement, or single measure of success for school effectiveness,

because each school operates in a unique context, with unique individuals – what works in one may

literally not work in another. Leaders therefore need to be able to respond appropriately to their

context as ‘designers of learning’ rather than ‘deliverers of pre-determined curricula’. In other

words they need to be able to lead by the spirit, not the letter of leadership for learning (Marshall

and Drummond, 2006).

Research into complex systems offers some signposts for this enquiry. There are properties of

complex systems that form a set of principles which underpin learning design. Blockley (Blockley,

2010) identifies 'layers, feedback loops and processes' as key properties and emergence as the

unpredictable outcomes of the relationships and interactions of key processes within a system.

Emergence is at the heart of complex systems thinking - thus the challenge of dealing with

uncertainty and risk is a challenge at the heart of leadership. Fundamentally this requires us to

recognise that there are limits to what we can know and therefore predict. Knowing what we don’t

know and acting accordingly is 'humility', a core virtue for both leaders and learners, for without

humility there can be little new learning or change. The ability to purposefully adapt and change

throughout a lifecycle is what makes an organisation or an individual resilient and sustainable. At

the heart of this resilience and sustainability is learning - self-aware, purposeful, conceptual

reorganisation - at all levels of the system: students, teachers, leaders and parents/carers.

In terms of developing evaluation models for schools, systems thinking also demonstrates that a

reductionist focus on the measurement and improvement of a single variable (for example a test

result) distorts both the process and the outcome of the system (James and Gipps, 1998); (James et

al., 2007);(James et al., 2007, Assessment Reform Group, 1999, Reay, 1999). As Mason (Mason,

2008) argues

'trying to isolate and quantify the salience of any particular factor is not only impossible, but also wrongheaded'. Isolate, even hypothetically, any one factor and not only is the whole complex web of connections among the constituent factors altered - so is the influence of (probably) every other factor too'. (2008:41)

22

We know for example, that an over focus on high stakes summative testing and assessment not

only distorts how teachers teach, but it also distorts student learning and creates an 'own goal'

since it depresses student motivation for learning (Harlen and Deakin Crick, 2003b, Harlen and

Deakin Crick, 2003a). It does more than simply relegating wider outcomes to second class goals – it

actually scores an own goal.

Not only is it important to focus on a range of processes and variables in schools which are deemed

to fulfill a particular purpose, but it is also important from a systems perspective to understand the

whole, the parts and how they interact. These are described by Goldstone (Goldstone, 2006) as

'contextualised' and 'decontextualised' aspects of a system. In developing contextualised accounts

learners and their environments (students, teachers, leaders and organisations) are seen as parts of

a single whole. How someone learns depends in part on the larger system in which they learn.

Elements of the system (both individual learners and other system elements) cannot be understood

independently. Rather, the interactions of the elements give rise to emergent behaviours that

would not arise through their independence.

A key concern with a singular focus on contextualisation is its inability to lead to generalization

(Goldstone, 2006). But complexity theorists have also identified some principles of complexity that

can be applied to different cases from seemingly unrelated domains - for example, Blockley's ideas

of layers, feedback loops and processes, and the ideas of emergence and uncertainty. Thus, while

learning, teaching and leadership are contextualised, we can also understand them as

decontextualised and identify patterns which can be generalized across highly contextualised

instances. So these two apparently contradictory ideas – contextualisation and decontextualsiation

– are two aspects of a common process of conceptual reorganisation - i.e, learning (Goldstone,

2006, p.37). In other words, students, teachers and leaders must recursively consider general

principles and specific contexts in order to learn.1.

UNDERSTANDING SCHOOLS AS COMPLEX SYSTEMS

We now turn to the model of a school as complex learning system which has informed this pilot

study. A system is defined by its purpose (Blockley, 2010, Blockley and Godfrey, 2000) and the

primary purpose of a school is the learning and achievement of its students. As we have argued,

learning is a core property of a resilient system. Thus for schools, learning is both a core process

and a product (desired outcome). This makes the application of systems thinking particularly salient

for schools - in contrast for example, to a supermarket in which learning is a core process but

providing food services and generating profits are the desired outcomes. In understanding the

defining importance of purpose in a system, we also foreground the concept of direction, of

journey of dynamic change and of lifecycle. For an individual learner, a team or an organisation, this

1 For a further glossary of terms from systems thinking which are relevant to schools see Appendix 1

23

implies an intelligent shared direction - and thus leadership. An individual student who is taking

responsibility for their own learning and life story is exercising personal leadership towards a

chosen purpose. A team or an organisation which is moving towards a shared purpose is exercising

leadership through individual, team and organisational learning and change. For a school as a

complex living system, leadership and learning are thus core processes as well as desired outcomes.

Drawing on (Goldspink, 2007), (Bryk et al., 2010), (Deakin Crick et al., 2011, Deakin Crick et al.,

2010, Deakin Crick, 2009), we have identified three key processes in schools as learning

communities, which constitute sub-systems or layers of learning and change processes which

provide 'viewpoints' from which to understand the system as a whole. These are (i) leadership -

including both community and school (ii) teacher professional learning and (iii) student engagement

in learning and achievement. In the next section we present the rationale for selecting these three

viewpoints.

Table 1 Three sub-systems or viewpoints for schools as complex systems

VIEWPOINT ONE: LEADERS LEADING LEARNING AND CHANGE IN THE COMMUNITY

In their conclusions to an extensive international survey of educational leadership and management

Davies and West-Burnham (2003) highlight several challenges for school leadership in the future,

including equity and entitlement, social trends, policy and innovation, funding and the nature of

‘schooling’ itself, how and where it happens and what it comprises. They go on to argue for new

models of leadership that are fundamentally concerned with strategy, values and learning. Beare

(2001), looking to the future, suggests that schools will need a focus on learning as the prime

mission and professional leadership where the leader and leadership teams give highest priority to

the professional purpose of the school, personally and frequently monitor the learning

programmes, put time and energy into school improvement, give support to the staff involved with

learning programme (and) put tangible emphasis on instructional leadership.

The relationship between school leadership and learning has been explored in several studies (e.g.

National College for School Leadership 2004; 2010) which all concluded that the most successful

Teachers

Leaders Students

24

systems, based on measures of student engagement and attainment, prioritised staff motivation

and commitment, teaching and learning practices and developing teachers’ capacities for

leadership (Gunter 2001; Bottery 2004). Cochran-Smith (2003) and Darling-Hammond and

Bransford (2005) and several other studies have established the importance of effective teaching

for supporting enhanced student achievements, and there is now a great deal of evidence behind

the claim that leadership that focuses on the quality of teaching is crucial for maintaining and

supporting improvement in the quality of learning in schools (Robinson, Lloyd and Rowe, 2008). At

the heart of leadership for learning (e.g., MacBeath and Cheng, 2008) is the concern with making

schools learning organisations supporting teachers’ and students’ learning and with greater

outreach to the communities they serve. And, as Silins and Mulford (2002) found in their

comprehensive study of leadership effects on student and organisational learning, student

outcomes are more likely to improve when leadership is distributed throughout the school

community and when teachers are empowered in their spheres of interest and expertise. The

McKinsey Report (2007), derived from an international survey of the most successful education

systems, found that a focus on teacher recruitment and professional learning were more important

as determinants of success than funding, social background of students, regularity of external

inspection or class sizes. However, PricewaterhouseCoopers’ (2007) study into school leadership

sounded a warning note when it found evidence that many school leaders, as a result of external

demands, felt that they did not have sufficient time to focus on teaching and learning.

In Bryk et al’s (2010) research, the most effective school leaders were catalytic agents for systemic

improvement, synchronously and tenaciously focusing on new relationships with parents and

community; building teachers’ professional capacity; creating a student-centred learning

environment and providing guidance about pedagogy and supports for teaching and learning.

Goldspink's (2007) research identified that the leadership qualities required for complexity are not

among the typical selection criteria for principals. They include: a level of modesty and

circumspection and a capacity to question one’s own deepest assumptions while inviting others to

participate in critical enquiry. These personal qualities and the assumptions about leadership as a

core systems process which underpin them have not been widely applied in education and few

school leaders are familiar with the relevant investigative, dispositional and analytical processes

(Zohar 1997).

Viewpoint Two: Teachers as learners

Collaborative, classroom-based, research-informed professional learning and enquiry in schools

represents an important facet of the ‘deep structures’ of learning in organizations and their

communities. Professional learning is a vital pre-condition for school improvement through its

positive influence on teachers’ classroom practices and their students’ learning (e.g., Birman, et al,

2000; Cohen and Hill, 2001; Day and Leith, 2007; Desimone et al., 2002; Garet et al., 2001; Loxley et

al., 2007). Schools’ promotion of teachers’ participation in school-to-school and other networking

activity can support improvement-related activity in schools and classrooms through enabling

25

teachers to access and engage with an expanded pool of practice ideas, resources, and sources of

support, increased opportunities for mutual problem-solving, knowledge creation and transfer, and

a heightened sense of valuing professional achievement and accomplishment (e.g., Little and

Veugelers, 2005; Jackson and Temperley, 2007; Stoll et al., 2007; Katz and Earl, 2010). Thus, an

excavation of a school’s deep learning structures is likely to reveal that at the heart of those

structures is the teacher as learner and scholar (Shulman, 2011).

There has been a great deal of international interest in understanding how professional learning

can fulfil its potential for supporting school improvement. For example this question has been

addressed by Meiers and Ingvarson (2005) in Australia; Piesanen, Kiviniemi, and Valkonen (2007) in

Finland; Timperley and Alton-Lee (2008) in New Zealand; and Garet, Porter, Desimone, Birman, and

Yoon (2001) in the USA. However, despite the importance of teachers’ learning, a great deal of

teachers’ professional learning activities are ineffective (Sykes, 1996; Hanushek, 2005). Borko

(2004), Clarke and Hollingsworth (2001), and Timperley and Alton-Lee (2008) have all argued that

the problem stems from simplistic conceptualisations of teachers’ professional learning that fail to

consider how learning is embedded in teachers’ professional work and the complex institutional

and community contexts in which that work develops. An important purpose of the proposed

research is to develop theoretical and empirical understandings of teachers’ learning and practice

as a complex system (Clarke and Collins, 2007; Collins and Clarke, 2008; Curtis and Stollar, 2002;

Davis and Sumara, 2006; Weaver, 1948) and to help schools develop appropriate strategies for

supporting its emergence and scaling up.

Misunderstanding the nature of teacher learning by underplaying its complexity leads to focus on

the micro-context (individual teachers or individual activities or programmes) to the exclusion of

influences from meso (institutional) and macro (school system) contexts (e.g., Bore and Wright,

2009; Bottery and Wright, 1996). Adopting a complexity thinking perspective, we assume that

teacher learning does not emerge as a series of isolated events but simultaneously in the activity of

autonomous entities (teachers), collectives (school phase and subject groups) and subsystems

within grander unities (schools within school systems within socio-political educational contexts).

These nested systems and subsystems associated with teacher learning are interdependent and

reciprocally influential. As a result, to explain teacher professional learning, one must consider what

sort of local knowledge, problems, routines, and aspirations shape and are shaped by individual

practices and beliefs. How are these then framed by the other systems involved? Thus we construe

teacher learning as a complex system representing recursive interactions between systems and

elements that coalesce in ways that are unpredictable but also highly patterned (Clarke and Collins,

2007). Therefore, identifying emergent patterns of interaction within and between levels of activity

that would constitute an explanatory theory, here, of teacher learning as a complex system

requires variable-inclusive (as opposed to control) strategies for research, development, planning

and evaluation.

26

Complex systems need to be off balance in order to move forward. Wheatley (1999) in her analysis

of large institutions as complex systems, notes that organizational equilibrium is “a sure path to

institutional death”. Helping schools and groups of teachers become aware of the full range of

dissonance between their values and practices in relation to teachers’ and school learning and

leadership was an effective intervention for promoting change, growth and deep learning as part of

the Learning how to Learn project (Pedder et al., 2005; Pedder, 2006; 2007; Pedder and MacBeath,

2008). Further mapping of patterns of dissonance in schools nationwide was an important feature

of the State of the Nation CPD study (Pedder et al., 2010; Opfer and Pedder, 2011). Dissonance

between what teachers consider important for enhancing the quality of their students’ learning

opportunities and perceptions of current practice may result in what Woolfolk et al (2009) refer to

as ‘change-provoking disequilibrium’, further underlining the practical significance of attending to

relationships between values and practices. Argyris and Schön’s (1996) work on ‘theories of action’

illustrate that dissonance serves as a catalyst for schools to attempt to change their environment in

ways that better support learning.

Viewpoint Three: Students as learners - deep learning as a complex psycho-social system

Deep learning occurs when students choose to invest in processes of learning that are authentic,

personally owned and enable agency in processes of knowledge construction. Learning Futures

research (2010, 2011) identified 'authenticity, agency and identity' as key elements of pedagogy

which lead to engagement and depth in learning, rather than superficial recall or performance

orientation. Where these occurred, students described their learning as transformative for them as

individuals - they were authors of their own learning journey, in a process of 'becoming' (Seely

Brown 2009). Engagement in learning is necessary for depth of outcome, but it is a complex

construct. Fredricks et al's (2004) review identified the components of engagement most

commonly identified by researchers as behaviour and participation, (Fullarton, 2002; Willms, 2003),

emotion - a sense of belonging and value (Willms, 2003) and cognitive beliefs about learning and

achievement (Munns and McFadden, 2000). A range of studies provided evidence about how

these variables contribute to engagement including: involvement and wellbeing (Zyngier, 2004;

Shernoff et al., 2003; Goldspink, 2008); interest (Hidi and Renninger, 2006; Eccles et al., 1998)

student epistemic assumptions, including assumptions about fixed/variable intelligence (self-

theories) (Dweck, 2000; Baxter Magolda, 2004; Cano, 2005; Hofer, 2001; Schommer, 1990); meta-

cognitive skill which has been found to have a greater impact on outcomes than intelligence

(Veenman et al., 2006); and students’ learning power (Deakin Crick et al 2010; Deakin Crick & Ren

2011; Deakin Crick 2011). In addition to factors internal to the student, several studies identify the

influence of pedagogy and school climate on learner engagement: school culture and climate

(Anderson, 1982) and quality pedagogy (Newmann and Wehlage, 1995; Ladwig and Gore, 200;

Ladwig et al., 2007; Hattie, 1999; Hattie, 2003). Some aspects of pedagogy are particularly

important to the engagement of low SES and Indigenous learners in Australia (Gale et al., 2008;

Amosa et al., 2007; Zevenbergen et al., 2004) as are aspects of learning power and self reflection

(Deakin Crick and Grushka, 2010; Deakin Crick, 2009c; Goodson and Deakin Crick, 2009). Fredricks

27

et al. summarised their review by suggesting that the individual types of engagement (behavioural,

cognitive, emotional) have 'not been studied in combination, either as results of antecedents nor as

influences on outcomes' and that research has tended to use variable-centred rather than pattern-

centred techniques, cross-sectional rather than longitudinal (2004, p.87). The result is that we have

little information about the interactions between different aspects of learner engagement and little

information about the development and malleability of engagement over time.

Central to engagement is the idea that the learners themselves need to want to learn, and to

become aware of themselves as learners. Black and colleagues (2006) argued that a focus on

learning to learn and assessment for learning in schools is important particularly when it leads to

the promotion of ‘autonomous learning’ or ‘intentional learning’ (Bereiter and Scardamaila 1989).

Intentional learning implies agency and choice on the part of the learner and goes beyond the

acquisition of study skills and strategies, requiring practices which invoke the learner to take

responsibility for their own learning in a relational context. Hautamaki et al., (2002) also emphasise

the importance of learner agency and self-regulation.

Learning power is a central concept in engagement. It is a term used to describe that complex mix

of personal qualities that characterise effective learners and is defined as ‘a form of consciousness

characterised by particular dispositions, values and attitudes, with a lateral and a temporal

connectivity’ (Deakin Crick 2007). Learning power operates through relationships, where trust,

affirmation and challenge play a constitutive role and it is ‘storied’ in the trajectory a person brings

to their learning and in their hopes and aspirations. Successive empirical studies identified seven

dimensions of learning power: changing and learning, critical curiosity, meaning making, creativity,

learning relationships, strategic awareness and resilience and a self-assessment tool based on these

scales, which provides a framework for a mentored conversation which moves between the

learner, their story and a negotiated learning outcome (Deakin Crick, 2007; 2005, Deakin Crick et

al., 2004).

What has also emerged from these empirical studies is that the self-assessment of learning power

is a necessary but not sufficient aspect of a learning ecology. Other key factors include (i) the

quality of learning/assessment relationships; (ii) the creation of a locally owned language for

learning, including dialogue; (iii) the extensive use of metaphor and modelling; (iv) the use of

learning power dimensions as scaffolding for enquiry, (v) the re-sequencing of the content of the

curriculum, and (vii) the relationship between the personal/autopoietic and the public in relation to

learning. Learning power is thus a part of a ‘learning journey’ set in a complex living system. Seely

Brown and Thomas (2009) suggest that this journey is about ‘learning to become’ set in a

participatory framework, which embraces embodied and tacit knowledge as well as explicit

knowledge (Polayni 1967; Heron and Reason (1997).

These are some of the complex conditions necessary for deep learning, embedded in a critical

socio-cultural context (Habermas 1972; Freire 1972) in which students identify value and purpose

and are authors of their learning, embodied in a particular context in place and over time (Goodson

28

and Beista, 2010, Goodson and Deakin Crick, 2009, Goodson, 2009). Deep learning is described by

Bateson (1972) as third level learning, which involves personal transformation – rather than only

repetition (primary learning) or learning to learn (secondary learning). Deep learning is best

understood as a complex psycho-social system, with subsystems which include identity, experience

and story. The learner arrives at a learning opportunity already possessing a way of knowing and

being in the world which is the sum of their experience to date. Vygotsky (1978, 1962/1934)

described this as ‘perezhivanija’, the term used for accumulated lived emotional experience,

including values, attitudes, beliefs, schemas and affect. For Vygotsky, perezhivanija is the process

through which interactions in the ‘zone of proximal development’ are perceived by the learner,

described by Mahn and John-Steiner (2002) as a complex whole, a system of systems which

includes the inter-related and interdependent elements of participants, environments, artefacts

(such as computers, or tools) and context. Sfard and Prusak, (2005) suggest that the notion of

identity - ‘collections of stories about persons that are "reifying, endorsable by others and

significant" is the missing link between learning and its socio-cultural context. The challenge in this

project is to develop forms of pedagogy which attend to the complexity of student engagement in

learning and the concomitant depth of process and outcome, including (i) the selfhood, identity

and story of the student in community and tradition (ii) the personal qualities necessary to engage

with new learning opportunities (iii) the co- construction of knowledge, skills and understanding

and (iv) the competence to engage with authentic learning in the real world (Deakin Crick, 2011,

Deakin Crick, 2009b, Deakin Crick, 2009a).

THE CHALLENGE OF COMPLEX DATA COLLECTION AND REPRESENTATION FOR SCHOOLS

These three core processes inform the design of this pilot study. The challenge is to generate a

systems design for a school and then evaluate and improve its core processes and their outcomes

through data collection, analysis and rapid feedback in a way which enables leaders to make

decisions based on a wide and complex range of evidence and provide performance data for

accountability purposes. This involves complex data collection, representation and interpretation

which offers a challenge to traditional approaches in which a ‘post positive scientific method’

encourages a reductionist focus on a part, not the whole. Leadership requires the harnessing of

collective intelligence – and the speed and complexity with which data can now be manipulated

and represented presents unique learning challenges for leaders. Understanding the weight of

evidence which such data offers is a further challenge.

Complexity in learning communities can be understood in different ways. A problem can be

complicated (i.e predictable) but not complex. Complexity can be highly complex (unpredictable

and uncertain) evidentially complex (obviously so) and causally complex (one thing leads to

another). Figure 2 reproduces Barr's (2013) typology of complexity, and overlays these types of

complexity onto some of the problems of leading learning communities. These problems are not

29

confined to any one part of this diagram. Whilst they seem to be more ‘complex’ than ‘complicated’

in nature, the distinction is still an important one and relates to both axes of complexity – that is,

the problems of learning communities are a mixture of ‘highly complex’, ‘evidentially complex’,

‘causally complex’ and (merely) complicated.

Table 2 A typology of complexity in learning communities

CHOICE OF APPROACH TO RESOLVING COMPLEXITY IN LEARNING COMMUNITIES

The debate in the domain of school improvement and effectiveness studies has already been

shown to be influenced by significantly different worldviews within educational research. Often the

post positivist world view is described as 'objective' whilst an interpretive worldview is described as

'subjective' and these are set in opposition to each other. Morgan and Smircich (1980) set out a

‘network of basic assumptions characterizing the subjective-objective debate within social science’.

Set out against a linear six-point scale of approaches to social science from ‘subjectivist’ to

‘objectivist’ these assumptions concern ontology, human nature and basic epistemological stances.

From this network, the assumptions that most closely match the complexities of learning

communities described previously are selected and presented in Table 1.

Table 3 Selection of assumptions appropriate to complex problems of learning communities

(with assumptions after Morgan and Smircich, 1980)

Type of

assumption

Descriptions appropriate to complex

problems of learning communities

Position on subjective-

objective spectrum

30

Core ontological

assumptions

‘Reality as a contextual field of information’

‘Each school operates in a unique context’

(Davis and Sumara, 2006)

‘Reality as a social construction’

‘Complex web of connections among the

constituent factors’ (Mason, 2008)

‘Deep learning as a complex psycho-social

system’ (Learning Futures, 2010)

4: Borderline objective/

subjective

2: Somewhat subjective

Assumptions about

human nature

‘Man as social constructor’

Complex mix of personal qualities that

characterise effective learners (Deakin

Crick, 2012)

2: Somewhat subjective

Basic

epistemological

stance

‘To map contexts’

In Bronfenbrenner (Bronfenbrenner, 1989)

terms ‘System of systems which includes

the inter-related and interdependent

elements’

‘To understand how social reality is created’ ‘Helping schools and groups of teachers

become aware of the full range of

dissonance between their values and

practices’ (Pedder and MacBeath, 2008)

4: Borderline objective/

subjective

2: Somewhat subjective

1 describes the position of each assumption against Morgan and Smircich’s (1980) six-point scale

from ‘subjective’ to ‘objective’. The pattern of these choices across the spectrum determine that a

multi-methodology (Mingers and Brocklesby, 1997) is needed for dealing with the diversely

complex problems of learning communities. Morgan and Smircich’s network of assumptions aligns

with the following research methods, which are subsequently considered in more detail for

inclusion in the research methodology:

Hermeneutic analysis of results of qualitative and narrative surveys of the experiences of

stakeholders including students, parents/carers, teachers and principals;

Historical quantitative analyses of schools performance data for the same students; and

Interpretive contextual analysis of experiences and performance data.

SYSTEMS THINKING FOR RESOLVING COMPLEXITY AND MANAGING UNCERTAINTY

Systems thinking offers a way of both getting to grips with complexity and a forum for engagement

and participation with the leaders, students and teachers. Systems thinking aims to overcome

traditional barriers to such development by seeking to make complex ideas and data accessible in

such a way that innovation and progress are possible. It does this by attending to systems design –

31

of an organisation or a process – and modelling that complexity through a visual analytic. This then

provides the salient parameters for a measurement model which informs intelligent leadership. `

Of the many systems thinking approaches available (Jackson, 2000; Rosenhead and Mingers, 2001),

the most appropriate for supporting collaborative development and leadership decisioning in

complex systems similar to learning communities, is hierarchical process modelling (eg Davis et al,

2010) which has three important characteristics:

Visual/ effective reporting of complex ideas and information is enhanced using hierarchical

mapping of processes and an ‘Italian Flag’ model of evidence;

Assimilating all forms of evidence – data, prediction and opinion; and

Facilitating access to key information required for informed discussion, innovation and

agreement.

The Perimeta software was developed by University of Bristol as a research tool for systems

thinking. It supports collaborative development of solutions to complex problems by providing a

highly visual interface for understanding complex cause-and-effect and complex evidence.

Commercial applications of Perimeta have been developed. Perimeta can be described as:

A learning analytic designed to model diverse and complex processes

Driven by stakeholder purpose

Capable of dealing with hard, soft and narrative data in evidence of success, failure and

‘what we don’t know’

A visual environment for sense-making in complexity

A framework for self-evaluation and dialogue

RESEARCH DESIGN AND METHODOLOGY

SYSTEMS DESIGNING AS IMPROVEMENT RESEARCH DESIGN

The approach to the research design and methodology for this pilot project followed the process of

systems designing developed in the Systems Centre at the University of Bristol (Blockley, 2010,

Blockley and Godfrey, 2000). The sequence of events for the pilot project are as follows: First, the

system boundaries of the OCL schools were defined according to their locally defined purpose.

Next a rich picture was elicited about the system, including identifying stakeholders and the core

processes for achieving the system’s purpose, together with their unique measurement

parameters. From here, key outcomes of the system which were deemed to fulfil its purpose

formed criteria against which the evidence gathering process was defined. These were designed for

entry into the Perimeta software for modelling. The software then took a range of types of data and

provided evidence about what is working well – what is working against desired outcomes and

32

what is not known. This is returned to the stakeholders as a rich, visual analytic to be used for

decision-making, celebration and improvement. What this approach offers for the challenge we

are addressing in this pilot project, is the ability to take the outcomes of traditional quantitative

data and qualitative data (including narrative) and combine it in a bespoke systems design to

visually re-present an overall picture of development and achievement against several key target

outcomes. The computation of traditional outcomes into the 'Italian flag' model of

certainty/uncertainty about success or failure and its visualisation allows leaders to apprehend a

large amount of complex data, encouraging holisitic thinking. It also permits the interrogation of

high level indicators of failure, or uncertainty in any core outcome through 'drilling down' into

much greater detail in order to aid leadership decisioning.

CASE STUDY SAMPLE

The sample chosen for this case study was a group of three Academies in the UK. Oasis Academies

are secondary schools in the United Kingdom, governed by Oasis Community Learning which is a

charitable trust. The vision of Oasis Academies ‘is to create both outstanding schools and

community hubs’. As described in its Education Charter Oasis (See Appendix 2) has a commitment

to ‘transforming lives, learning and communities to achieve stated outcomes for students, staff,

parents/ carers and community members’. A continuing challenge for Oasis is twofold: firstly, how

to evaluate these wider outcomes of schools; and secondly where to target improvements to best

effect. This project is referred to within the Academy chain as ECHO - Evaluating Charter

Outcomes.

Academic results are important but other, less easily quantifiable measures of success are relevant

to transforming lives, learning and communities. For example, the development of students as life-

long learners, employability skills, citizenship, self-confidence, teamwork and emotional wellbeing

are widely recognised as essential qualities for individual success in adult life and for social

cohesion. In the UK, OFSTED2 inspections continue to monitor broader aspects of education like

spiritual, moral, social and cultural development but this evidence typically becomes secondary,

rather than integral, when overall and final judgements are made about performance.

This sample was chosen because of the shared commitment of the Trust to develop useful ways of

assessing complex processes in schools and their communities which encourage powerful learning

and feedback at all levels.

PHASING OF RESEARCH

A phased approach to the research project was adopted, illustrated in 3, with the twin aim of

testing the model and of providing evidence for improvement for Academy leaders and the Trust.

The overall project proceeded in five key steps. First, the designing process, which entailed

2 Office for Standards in Education, Children’s Services and Skills

33

identifying the vision, core processes and key outcomes, specific to this group of Academies. It also

included the identification of the measurement model. The second step was the collection of data

to meet the measurement model designed to represent progress towards the key outcomes. Step

three was the construction of the Perimeta HPM Model. The aim of these first three steps was to

build understanding about the complexities of processes and evidences relating to learning

communities and to collect and analyse data so as to test the efficacy of the model. Step four was

feedback to users for leadership decisioning and exploration of solutions and, for the Oasis

Community Learning Acadamies, step five involves embedding the new practices.

Since this was a pilot project the generation of the Perimeta model was a lengthy process which

took place over a school year. This report focuses on this part of the project, with key areas for

development reported in the findings section. Because it was a pilot project the data were

analysed traditionally and within Perimeta and comparison and triangulation of the results was an

important part of the process. This report is available for the professional communities concerned

to evaluate as part of the overall pilot.

Table 4 Phasing of research

STEP ONE: DESIGN MODEL AND MEASUREMENT PARAMETERS

UNDERSTANDING STAKEHOLDER OBJECTIVES AND CONSTRAINTS

Four main groups of stakeholders in Oasis Academies are recognised in the context of a learning

community:

Students (with Year subgroups);

Parents/ carers of students;

Teachers; and

Senior leaders.

The objectives of stakeholders are complex at an individual/ personal level and also at the collective

level (for example Year groups, the teaching staff). In addition to the standard measures of schools

performance in the form of KPIs and GCSE results, the interwoven Strands of the ECHO research

were designed to address a number of critical success factors founded on validated research. These

critical success factors are summarised in Table 2 below.

Design Model & Measurement

parameters

Collect and analyse all data

Generate and analyse Perimeta

model

Feedback to leaders & explore

solutions

Embed new practices

34

Table 5: Critical success factors for stakeholders

Critical success factors Data Collection Strands Staff

Students and staff feel valued and included in a community that is focused on learning and achievement

Strand 6 Teacher questionnaire Strand 2 Students Questionnaire Strand 3 student narrative interviews

Students and staff are effective learners and believe they are being helped to reach their full potential

Strand 1 ELLI questionnaire

Students enjoy a wide range of opportunities for learning and believe that broad aspects of their development are being catered for

Strand 2 Students Questionnaire Strand 3 student narrative interviews

Students maintain a good rate of progress at 3 key transitions - 6/7, 10/11 and 13/HEI

Strand 9 Attainment data for Year 11 Strand 4 Y11 Questionnaire

Students and staff achieve results that meet aspirations and expectations and feel that their successes are rewarded

Strand 6 Teacher questionnaire Strand 8 KPIs student progress

Students and staff contribute actively to a community that focuses on service to others and where there is shared leadership

Strand 6 Teacher questionnaire Strand 3 student narrative interviews

Staff can provide evidence that training and development opportunities have helped them to improve their classroom practice and/or their effectiveness as leaders and managers

Strand 6 Teacher questionnaire

Staff contribute actively to a community that learns together, shares what works best and knows what to do to achieve

Strand 6 Teacher questionnaire

Parent's/carers feel included and involved in their children’s education at the Academy and know there is an open door for contact when they need it

Strand 5 Parents/carers questionnaire

Senior leaders are effective role models as 'leaders of learning'

Strand 7 Leaders questionnaire

Constraints in the achievement of stakeholder objectives were not pre-supposed but rather tested

through analysis of the responses to questionnaires and interviews. A number of constraints

became evident in the responses and are discussed in more detail in Section Error! Reference

source not found.. In subsequent Phases of the ECHO project the learning about constraints will

enable objectives to be refined so that they are more specific to the stakeholders of Oasis

Academies.

UNDERSTANDING STAKEHOLDER EXPECTATIONS

As with the treatment of constraints, expectations of stakeholders regarding their performance

were not pre-supposed but rather tested through analysis of responses. Each questionnaire

included a number of statements designed to elicit self-perceptions of the adequacy of individual

performance, for example ‘I did as well as I had hope in my GCSEs’. Expectation of performance was

35

therefore treated as an output of the Integrated Approach, that is as a property of the learning of

stakeholders.

The selection of process performance measures for the analysis of responses (see (d) below) was

therefore separated from any consideration of constrains or expectations.

MAPPING CAUSE AND EFFECT

Figure 4 illustrates the alignment of high-level processes contributing to a learning community.

Although incomplete, this provides a systems overview of how an Academy operates in pursuit of

stakeholder objectives. The highest level represents the aims of society in general, and of

communities, regarding the education of children. The next level represents the outcomes achieved

through education by individual stakeholders including students but also their parents/carers,

teachers and leaders. The third level in this illustration represents the outputs of education in the

form of self-perceptions of learning by individuals in each of these groups. In the fourth level the

outputs of specific tests are related to third-level perceptions of performance.

Table 6 Illustrative performance hierarchy for a learning community

The Oasis Charter provided the majority of the first three levels including the vision of a learning

community and critical success factors. From the Transforming Learning section of the Charter,

thirteen statements were articulated to describe the intended behaviours (inputs) and experiences

of students, teachers, leaders and parents/carers, creating a set of critical success indicators for

Oasis Academies. Table 7 develops the illustrative systems view of Figure 4 into a hierarchy of

processes for a learning community based on the Oasis Charter.

Table 7 Hierarchy of specific requirements for success of a learning community based on the Oasis Charter

Level 1:

Vision for

learning

community

1.1 Establishing and sustaining a group of high achieving learning communities that enables

everyone to realise their full potential and refuses to put limits on achievement

Level 2:

Stakeholder

2.1 Developing the

learning of students so

2.2 Developing the

learning of teachers so

2.3 Developing the

learning of leaders so

2.4 Engaging

parents/carers

36

outcomes that they realise their

full potential

that they realise their

full potential and lead

the learning of

students effectively

that they lead the

learning of teachers

and students

effectively

effectively in the

learning activities of

the Academy and in

supporting the

learning of their

children

Level 3:

Outputs of

key

processes

3.1

Fee

ling

va

lue

d a

nd

in

clu

ded

in o

ur

com

mu

nity

3.2

Fee

ling

pa

rt o

f a

co

mm

unity

tha

t fo

cuses its

activitie

s o

n

lea

rnin

g a

nd

achie

ve

me

nt

3.3

Kn

ow

ing

th

at I

am

becom

ing

an

effe

ctive le

arn

er

3.4

Kn

ow

ing

th

at

we

are

be

ing

he

lped

to r

each

ou

r fu

ll po

tentia

l

3.5

Ma

inta

inin

g a

goo

d r

ate

of

pro

gre

ss a

t ke

y tra

nsitio

ns,

for

exam

ple

, p

rim

ary

to s

eco

nd

ary

scho

ol

3.6

Ach

ievin

g r

esu

lts th

at m

eet

my a

spir

atio

ns a

nd e

xp

ecta

tion

s

3.7

Fee

ling

tha

t m

y s

uccesse

s

are

reco

gnis

ed

3.8

Co

ntr

ibutin

g a

ctive

ly t

o a

co

mm

unity w

hic

h f

ocuse

s o

n

se

rvic

e t

o o

the

rs

3.9

Co

ntr

ibutin

g a

ctive

ly t

o a

co

mm

unity w

he

re s

hare

d

lea

de

rship

is p

rom

ote

d

3.1

0 P

rovid

ing e

vid

en

ce t

ha

t

tra

inin

g a

nd

de

ve

lopm

en

t

op

po

rtu

nitie

s h

ave

he

lpe

d m

e to

imp

rove

my c

lassro

om

pra

ctice

an

d/o

r m

y e

ffe

ctiven

ess a

s a

lea

de

r a

nd

man

age

r

3.1

1 C

on

trib

uting

active

ly t

o a

co

mm

unity t

ha

t le

arn

s to

geth

er,

sh

are

s w

hat

wo

rks b

est a

nd

kn

ow

s w

ha

t it n

ee

ds t

o d

o to

imp

rove

3.1

2 F

ee

ling

in

clu

de

d a

nd

involv

ed

with

my c

hild

ren’s

ed

uca

tion

at th

e A

ca

de

my a

nd

kn

ow

ing

tha

t th

ere

is a

n o

pe

n

do

or

for

con

tact

wh

en

I n

ee

d it

3.1

3 B

ein

g e

ffective

ro

le m

od

els

as le

ad

ers

of

lea

rnin

g

Level 4:

Key

processes

Student experiences of learning

Teacher experiences of learning

Leader experiences of learning

Parent/ carer experiences of learning

STEP TWO: COLLECT AND ANALYSE ALL DATA

Data collection methods were then designed to collect evidence in each of these nines Strands

which mapped onto the level 3 outputs. These included closed questionnaires which combined

research-validated scales from previous studies plus bespoke questions designed to elicit evidence

about some of the specific outcomes of the Charter. There were semi-structured interviews and

narrative interviews with teachers, students and leaders. Additional standard key performance

indicators (KPIs) were collected from each Academy to cover attainment and attendance, behaviour

etc. These data collection methods were designed specifically to provide evidence about one or

more of the twelve input statements in the model. For an example of this mapping process see

Table 5 below. For a full report on all questionnaires and interview transcripts see Appendix Two.

During the pilot project, three Oasis Academies were involved in providing data:

From students in different Year Groups: questionnaires and individual interviews

From teachers: questionnaire and individual interviews

From senior leaders: questionnaire and recorded group discussion

From parents/carers: questionnaire

Other aggregated data that is publically available like attendance figures and exam results

4 summarises the nine Strands and sources of evidence used in the ECHO project.

37

Table 8 ECHO sources of evidence

ECHO

Strand

Stakeholder group Source of evidence (and planned number of respondents)

1 Students in Year 7 Research-validated questionnaire completed at the beginning and end of Year 7 (N =

450)

2 Students in Years 8

and 10

Questionnaire with some research-validated questions plus questions relating to the

Charter (N=600)

3 Students in Year 9 Recorded interview based on a few questions designed to get under the surface of

learning (N= 30)

4 Students in Year 11 Questionnaire designed to provide evidence about the impact of education at an Oasis

Academy on the transition to FE, training and/or work and on into later life (N=300)

5 Parents/ carers Questionnaire about engagement with and support from the Academy for children’s

learning (N=100)

6 Teachers Questionnaire and recorded interview about the impact of CPD on classroom practice

and experience of the Oasis ethos (rhetoric or reality?) (N=30)

7 Leaders Questionnaire for senior leaders initially, followed by discussion at a senior leadership

team meeting about their role as ‘leaders of learning’, their impact on the culture and

climate of the Academy and the learning of teachers and students (N=25)

8 - Data from Key Performance Indicators, e.g. attendance, exclusions, student progress

9 Students in Year 11 GCSE results (N=300)

STRANDS OF DATA

Each strand of data related to one or more of 12 critical success indicators (CSIs See Table 5.). In

identifying the following strands of data, it was important to be conscious of the overall context of

the Academy as a complex system and learning organisation. Boundaries of the ‘system’ are drawn

and defined by its purpose and its complexity is understood through the concept of layers and

feedback loops. The layers identified were the students, teachers, leaders, parents/community and

the feedback loops constitute learning as a core process. A key concept informing the design is all

individuals as effective learners.

STRAND ONE: DEVELOPMENT OF STUDENTS AS LEARNERS IN YEAR 7

This consisted of data obtained from the Year 7 cohort from all three Academies (N=450), using the

research-validated questionnaire Effective Lifelong Learning Inventory (ELLI), completed at the

beginning and end of Year 7. ELLI is a 72 item questionnaire measuring seven dimensions of

learning power. Available online, this instantly produces a profile of each learner, in the form of a

7-spoked spider diagram. This feedback then becomes the starting point for interventions such as

mentoring conversations and strategies for developing learning power, individually and collectively.

38

The data for input, therefore, was either 72 variables with values ranging from 1-4 (where 1 = a

little like me and 4 = not at all like me), or seven variables as a percentage score. The latter was a

more powerful indicator in social science statistical terms. In addition, the degree of change (from

pre-test to post-test) as a new variable could also be identified.

STRAND TWO: STUDENT QUESTIONNAIRE AND PROPOSED ANALYSIS

Data from a 40 item questionnaire with some research-validated items plus additional questions

relating to the Oasis Charter, answered on a scale of 1-4 was used with students from Years 8 and

10 (N=600). These were input into the Perimeta model as 40 variables with values ranging from 1-

4, (where 4 = very good and 1= poor). Additionally, a basic statistical computation was conducted

on the items to explore whether there were overarching themes in the data. This would reduce the

data to a set of more powerful variables as percentage.

STRAND THREE: STORIES OF TRANSFORMATION

This Strand involved interviews with students in Year 9 (N=30) based on questions designed to elicit

the quality of their learning. Video and audio recordings of students’ stories formed the data sets

and were facilitated by a teacher with four or five key questions relating to one or more of the CSIs.

The video recordings were thematically analysed and rated on a score of 1-4 (with 4 = very positive

and 1 = very negative).

STRAND FOUR: POST-16 TRANSITION AND PROGRESS TO ADULTHOOD

This Strand included a questionnaire for Year 11 from two of the Academies (N=300) which was

designed to provide evidence about the impact of education at an Oasis Academy on the transition

to further education, training and/or work and on into later life. The questionnaire contains 40

items, focusing on experiences at the Academy and the transition to the next stage. The aim was to

continue to track the cohort over a number of years.

These data were input as 40 variables with values ranging from 1-4 (with 4 = very good and 1=

poor). Additionally, basic statistical computations were conducted on the items to explore whether

there are overarching themes in the data – i.e. to reduce the data to a set of more powerful

variables as a percentage

STRAND FIVE: LEARNING IN A WIDER CONTEXT – QUALITY OF RELATIONSHIPS WITH

PARENTS/CARERS

This Strand involved a 40 item questionnaire for parents and carers (N=100) from one Academy

about their engagement with and support from the Academy for their children’s learning. As with

Strand Two, this questionnaire contained additional open-ended questions relating to the Oasis

Charter, answered on a scale of 1-4.

39

These were input as 40 variables with values ranging from 1-4, (where 4 = very good and 1= poor).

Additionally, a basic statistical computation was conducted on the items to explore whether there

were overarching themes in the data – i.e. to reduce the data to a set of more powerful variables as

percentage.

STRAND SIX: THE LEARNING OF TEACHERS - IMPACT OF CPD ON THE QUALITY OF CLASSROOM

PRACTICE

This Strand has a focus on teachers (N=30) with data collected via a three part 66 item

questionnaire about the impact of continued professional development on classroom practice and

experience of the Oasis ethos (rhetoric or reality) against the Oasis Charter on a four point Likert

type scale.

As with Strand Two and Five these were input as 66 variables with values ranging from 1-4, (where

4 = agree strongly and 1= disagree strongly). Additionally, a basic statistical computation was

conducted on the items to explore whether there were overarching themes in the data – i.e. to

reduce the data to a set of more powerful variables as percentage.

STRAND SEVEN: THE LEARNING OF PRINICIPALS – STRATEGIC APPROACHES TO THE

TRANSFORMATION OF LEARNING

The data for this strand came from questionnaires for senior leaders initially about their role as

‘leaders of learning’, their impact on the culture and climate of the Academy and the learning of

teachers and students (N=25 for senior leadership teams)

The questionnaire contains 55 items, including one open-ended question. The 54 closed items were

structured with a four-point Likert type scale (where 1 = strongly disagree and 4 = strongly agree).

These were inputted as 54 variables with values ranging from 1-4, as above. Additionally a basic

statistical computation was conducted on the items to explore whether there were any overarching

themes in the data – i.e. to reduce the data to a set of more powerful variables as a percentage.

STRAND EIGHT: CONTEXT FOR STUDENT LEARNING - KEY PERFORMANCE INDICATORS

Data from Key Performance Indicators, for example, attendance, free school meals (FSM)

exclusions, special educational needs (SEN) was used for this strand. There were up to 10

aggregated variables for each year group (N=5) and data was explored for differences within and

between Academies with regards to gender, FSM and SEN.

STRAND NINE: KEY OUTCOME FOR YEAR 11 STUDENTS – GCSE/EXAM RESULTS

Data for this Strand came from Year 11 GCSE results. Five aggregated variables will be used for

including 5+ higher GCSEs including English and Mathematics and Science. The data will be

40

explored for differences within and between Academies with regards to gender, FSM and SEN

against GCSE exam results in the key subjects.

MAPPING THE DATA STRANDS ONTO THE CRITICAL SUCCESS INDICATORS

The table in the next section presents the data map of each Strand of data by item, to each critical

success indicator in the model. This mapping was created and moderated by two researchers

forming and moderating judgements about data structure. This details how all the CSIs are

addressed within the nine Strands of data collection, although only examples are used because of

the size of the document.

41

Table 9 Critical Success Indicators mapped onto data collection strands

CRITICAL SUCCESS FACTORS

Data Collection Strands Staff

Data Collection Strands Students

Items or scales Data Structure

Focus of items or interviews for each CSI - assumptions and evidence for each CSI

Key issues or questions

Students and staff feel valued and included in a community that is focused on learning and achievement

Strand 6 Teacher questionnaire

Strand 2 Students Questionnaire Strand 3 student narrative interviews

Involvement in learning I was concentrating and hard to distract I took a lot of care with what I was doing I was working hard on the learning I was very focused on the learning Negative affect I was nervous I told myself 'I can’t do this' and felt unhappy I was afraid in case I got things wrong I was unhappy with what I did

Likert type scale

Students and staff are effective learners and believe they are being helped to reach their full potential

Strand 1 ELLI Strand 1 ELLI Changing and learning, meaning making critical curiosity, creativity, strategic awareness, learning relationships, resilience 72 items Openness to learning I did more than I was asked to do I came up with new ideas on my own When I found something hard I tried another way I was excited to explore new things Interest The subject we were doing is very interesting to me I have always been curious about what we were learning I found the subject boring (inverted) I could never be bored with this subject.

Percentage scores from scales of items Likert type scale

The quality and strength of each individual’s orientation towards learning - how open are they to engaging with new learning opportunities within the Academy? How well do they understand themselves as learners? How strong is their learning identity? Are they open to new experiences? Does what they learn hold their interest?

A two-concept CSI

42

CRITICAL SUCCESS FACTORS

Data Collection Strands Staff

Data Collection Strands Students

Items or scales Data Structure

Focus of items or interviews for each CSI - assumptions and evidence for each CSI

Key issues or questions

Students enjoy a wide range of opportunities for learning and believe that broad aspects of their development are being catered for

Strand 2 Students Questionnaire Strand 3 student narrative interviews

Positive Affect I felt proud of what I achieved I was very happy with what I did I felt content with my learning

Likert type scale Students feel good about being in school

A two-concept CSI

Students maintain a good rate of progress at 3 key transitions - 6/7, 10/11 and 13/HEI

Strand 9 Attainment data for Year 11 Strand 4 Y11 Questionnaire

Schools data sets Predetermined How can we measure progress at Years 6/7 and Year 13/HEI transition?

Students and staff achieve results that meet aspirations and expectations and feel that their successes are rewarded

Strand 6 Teacher questionnaire

Strand 8 KPIs student progress

A value added score - a regression co-efficient

A two-concept CSI

Students and staff contribute actively to a community that focuses on service to others and where there is shared leadership

Strand 6 Teacher questionnaire

Strand 3 student narrative interviews

Negative social functioning What I did upset others I played around instead of learning I did not do what was asked I did not want to work with others Positive social functioning I offered to help others I was included by other students I worked with others wherever I could

Likert type scale Students feel good about being in school

A two-concept CSI

Staff can provide evidence that training and development opportunities have helped them to improve their classroom practice and/or their effectiveness as leaders and managers

Strand 6 Teacher questionnaire

Can you give me an example of a time when…..

Narrative analysis as rating

43

CRITICAL SUCCESS FACTORS

Data Collection Strands Staff

Data Collection Strands Students

Items or scales Data Structure

Focus of items or interviews for each CSI - assumptions and evidence for each CSI

Key issues or questions

Staff contribute actively to a community that learns together, shares what works best and knows what to do to achieve

Strand 6 Teacher questionnaire

Teacher Enquiry Staff draw on good practice from other schools as a means to further their own professional development Staff read research reports as one source of useful ideas for improving their practice Staff use the web as one source of useful ideas for improving their practice Students are consulted about how they learn most effectively Staff relate what works in their own practice to research findings Staff modify their practice in the light of published research evidence Staff carry out joint research/evaluation with one or more colleagues as a way of improving practice Building social capital Staff regularly collaborate to plan their teaching If staff have a problem with their teaching they usually turn to colleagues for help Teachers suggest ideas or approaches for colleagues to try out in class Teachers make collective agreements to test out new ideas Teachers discuss openly with colleagues what and how they are learning Staff frequently use informal opportunities to discuss how students learn Staff offer one another reassurance and support Teacher Critical and responsive learning Staff are able to see how practices that work in one context might be adapted to other contexts Staff reflect on their practice as a way of identifying professional learning needs Staff experiment with their practice as a conscious strategy for improving classroom teaching and learning Staff modify their practice in the light of feedback from their students Staff modify their practice in the light of evidence from self-evaluations of their classroom practice Staff modify their practice in the light of evidence from evaluations of their classroom practice by managers or other colleagues

Likert type scale 1. Enquiry: using and responding to different sources of evidence, carrying out joint research and evaluation with colleagues 2. Teacher Critical and responsive learning, through experimentation, responding to feedback self-evaluation and reflection 3. Building social capital, learning, working and supporting each other 4.Teachers value learning

44

CRITICAL SUCCESS FACTORS

Data Collection Strands Staff

Data Collection Strands Students

Items or scales Data Structure

Focus of items or interviews for each CSI - assumptions and evidence for each CSI

Key issues or questions

Parent's/carers feel included and involved in their children’s education at the Academy and know there is an open door for contact when they need it

Strand 5 Parents/carers questionnaire Likert type scales A two-concept CSI

Senior leaders are effective role models as 'leaders of learning'

Strand 7 Leaders questionnaire

Likert type scales

45

STEP THREE: GENERATE AND ANALYSE PERIMETA MODEL

DEFINING PROCESS PERFORMANCE MEASURES

Performance measures were established for each of five types of processes of a learning

community. The five types were selected to allow a breakdown by some of the main areas of

interest regarding learning community performance, that is, by gender and by Academy. Other

areas such as the influence of social factors on performance will be considered for inclusion in later

Phases of the ECHO project. Table 6 summarises the objectives, measures and target performance

levels of the five types.

Table 10 Process performance measures

Type of process Objective Measure Target

Outputs:

Overall All stakeholders strongly agree with all

statements

Percentage

score

100% strongly agree

Gender All stakeholders of given Gender

strongly agree with all statements

Percentage

score

100% strongly agree

Academy All stakeholders of given Academy

strongly agree with all statements

Percentage

score

100% strongly agree

Question All stakeholders strongly agree with

given statement

Percentage

score

100% strongly agree

Inputs:

Responses to

statements in

questionnaire

Stakeholder strongly agrees with given

statement

Percentage

score

100% strongly agree

COMMUNICATING EVIDENCE

Two aspects of the performance of a learning community need to be communicated effectively in

order to make sense of the evidence and to support improvement. The first aspect relates to the

higher-level questions about how well the system is working, and the second relates to the qualities

of evidence derived from responses to questionnaires and interviews.

A simple dashboard was devised to assist in communicating evidence of performance in respect of

the many and diverse outcomes and outputs of a learning community. shows the four main

components of the dashboard, reflecting the following hierarchy of dimensions of performance of a

learning community:

By Question {Q1..Qj} answered by each Respondent {N1..Ni};

By Gender {G1, G2} for all Questions answered by Respondents of that Gender;

46

By Academy {A1, A2, A3} for all Questions answered by Respondents belonging to that Academy; and

For all Questions answered by all Respondents.

Table 11 Components of performance dashboard for a learning community

Performance by ACADEMY {A1, A2, A3} Performance OVERALL

Inputs by RESPONDENT and QUESTION {N1xQ1 .. Ni x Qj}

Performance by QUESTION {Q1 .. Qj}

Performance by GENDER {M, F}

Recognising the uncertainty inherent in responses to a questionnaire, the ECHO project adopted

the ‘Italian Flag’ graph to represent the quality of all of the evidence and consisting of:

‘Green’ representing the strength of evidence of positive self-perception of learning;

‘Red’ representing the strength of evidence of negative self-perception of learning; and

‘White’ representing lack of evidence, or uncertainty in self-perception of learning.

Table 12 illustrates Italian Flags for some examples of self-perception of learning.

Table 12: Illustration of Italian Flag graphs in context of a learning community

Categories of evidence in Italian Flag Evidence of success

Lack of evidence

Evidence of failure

Application to learning community Evidence of positive self-perception of learning

Uncertainty in self-perception of learning

Evidence of negative self-perception of learning

Example: High self-perception of learning from high proportion of ‘strongly agree’ responses

Example: Medium self-perception of learning from high proportion of ‘agree’ responses

Example: Medium self-perception of learning from high proportion of ‘disagree’ responses

Example: Low self-perception of learning indicated by high proportion of ‘strongly disagree’ responses

ELICITING EVIDENCE

The design of questionnaires and interviews was based on the systems view of a learning

community represented by Table 5. Nine Strands of evidence of performance relating to self-

perceptions of the experiences of learning of students, teachers, leaders and parents/carers was

47

then gathered for input to the model using questionnaires, semi-structured interviews combined

research-validated questions with additional questions about the specific outcomes of the Charter

and standard KPIs. The evidence provided by responses to each question or by themes from

qualitative or narrative interview data was specifically designed to provide evidence about one or

more of the thirteen input statements in the model.

Uncertainty is introduced in the evidence from questionnaires and interviews by answers that are

not absolutely positive or absolutely negative. Given a scale from ‘strongly disagree’ to ‘strongly

agree’, the intermediate points of ‘disagree’ and ‘agree’ entail some contrary evidence and some

uncertainty, as indicated by the examples given in Table 6.

Another factor in making sense of the evidence is the degree of confidence we have in the

individual respondent. A number of considerations are relevant, for example the respondent’s

understanding of the question and experience to answer it. The respondent may be considered to

be biased or in some other way misleading or misled.

Hall et al (2004) describe a technique for mapping from linguistic descriptions of ‘performance’ and

‘confidence in judgement of performance’ to interval values of performance compatible with Italian

Flag figures of merit. Table 13 reproduces their illustration of the mapping and presents an

interpretation of Hall et al’s mapping for a set of 25 discrete combinations of performance and

confidence scales. The performance scale is based on a four-point Likert scale from ‘strongly

disagree’ to ‘strongly agree’, corresponding to scores from 1 to 4 recorded by respondents to ECHO

project questionnaires.

Table 13 Hall et al’s (2004) mapping from linguistic descriptions of ‘performance’ and ‘confidence in judgement of performance’ to interval values

of performance

48

Table 14 Conversion of scores from Likert scale to Sp and Sn values of evidence

Confidence

judgment

Rating

Very high 100% (0.00,0.00) (0.25,0.25) (0.50,0.50) (0.75,0.75) (1.00,1.00)

High 75% (0.00,0.05) (0.20,0.30) (0.42,0.58) (0.70,0.80) (0.95,1.00)

Medium 50% (0.00,0.10) (0.10,0.40) (0.38,0.62) (0.60,0.90) (0.90,1.00)

Low 25% (0.00,0.25) (0.05,0.50) (0.30,0.70) (0.50,0.95) (0.75,1.00)

Very low 0% (0.00,0.60) (0.01,0.80) (0.05,0.95) (0.20,1,00) (0.40,1.00)

Rating 0% 25% 50% 75% 100%

Performance judgment 0 1 – strongly

disagree

2 - disagree 3 - agree 4 – strongly agree

MODELLING PROCESS PERFORMANCE

Process performance functions were created by combining the Likert conversion of Table 147 with

the process performance measurement scales of Figure 5. At the input level, direct evidence of

respondents’ raw scores (from 1 to 4) for each question were converted into Italian Flag figures of

merit where the best possible (100%) performance was full Green (respondent strongly agrees, and

very high confidence in the respondent) and the worst possible performance was mostly Red

(respondent strongly disagrees, but very low confidence in the respondent). For output processes

(collated by Question, by Gender, by Academy and Overall) the definitions of best and worst

performance and all points in between were judged on a similar scale.

MODELLING THE SIGNIFICANCE OF INTER-PROCESS RELATIONSHIPS

As established earlier, the significance of pairwise relationships or links between ‘parent’ and ‘child’

parameters/ processes is modelled in the Integrated Approach using three detailed attributes of

Sufficiency, Necessity and Dependency defined by Davis & Fletcher (2000):

‘The Sufficiency or relevance of the evidence to its parent process is judged as a single

number in the [0,1] range;

A sub-process is a Necessity if the parent process cannot succeed without it. Consequently, in

the event of failure of the sub-process, the parent process fails’; and

Dependency is the degree of overlap between sub-processes and describes the degree of

commonality in the sources of evidence’.

49

The significance of pairwise relationships or links between processes defining ‘causes’ and ‘effects’

was modelled using three detailed attributes of Sufficiency, Necessity and Dependency defined

earlier. On the basis of experience in modelling ‘many to one’ performance relationships, the values

indicated in Table 13 were assigned.

Table 15 Sufficiency, Necessity and Dependency of relationships between processes

‘Effect’ process ‘Cause’ process Sufficiency Necessity Dependency

Overall performance Responses to statements in

questionnaires

0.3 0.4 1.0

Gender performance Responses to statements in

questionnaires

0.3 0.4 1.0

Academy performance Responses to statements in

questionnaires

0.3 0.4 1.0

Question performance Responses to statements in

questionnaires

0.3 0.4 1.0

ESTIMATING SYSTEM PERFORMANCE

Full system models of the features of each of Strands 1 to 7 were developed in the Perimeta toolkit,

combining the following features defined above:

A hierarchy linking the responses to questionnaire statements (input processes) in turn to

output processes representing the performance by Question, by participating Academies, by

respondent Gender (where given) and Overall;

Responses to questions using Likert ratings from 1 (‘poor’ or ‘strongly disagree’) to 4 (‘very

good’ or ‘strongly agree’);

Process performance functions using linguistic measures related to Likert rating scales; and

Sufficiency, Necessity and Dependency ratings for each cause and effect relationship

The Perimeta models used the ‘Juniper’ algorithm to propagate the evidence and provide estimates

of output performance by Question, by Gender, by Academy and Overall. A full explanation of the

Juniper algorithm is given in the Appendix 6, and is based on the description given by Davis and

Fletcher (2000).

The Perimeta models were each configured to produce a dashboard summary view as well as

tabulated results for each Question, each Academy, each Gender and Overall. A sample datasheet

from a Perimeta model is reproduced in Table 16 below.

50

Table 16 Sample of Perimeta model datasheet

VALIDATING INSIGHTS

The parallel analysis by the ECHO project team of the response data using standard statistical

techniques allowed the results of the Integrated Approach to be validated. There following

comparisons were made between the two sets of results, for each Strand where appropriate:

Positive self-perception of learning, versus mean scores;

Strength of positive or negative bias, versus range of scores;

Uncertainty in self-perception of learning, versus standard deviation of scores;

Comparison of results between Academies; and

Comparison of results between Genders.

STEP FOUR: FEEDBACK TO LEADERS AND EXPLORE SOLUTIONS

Stakeholder validation was limited in Phase 1 of the ECHO project by lack of direct engagement

with stakeholders. However, detailed reviews were carried out twice during Phase1 with senior

leaders, providing valuable feedback and endorsement of the approach.

Step Four of the ECHO project would develop the approach to creating learning communities,

focusing on:

Deepening understanding of the issues of evaluation and refining questionnaires;

51

Developing the Integrated Approach and its application to the complex problem of creating

learning communities; and

Engaging stakeholders in developing solutions.

STEP FIVE: EMBED NEW PRACTICES

Step Five of the ECHO project would be a continuous process of creating learning communities,

transferring capability to the stakeholders and focusing on:

Supporting stakeholder-led improvements in learning with appropriate access to

appropriate tools and techniques;

Developing the Integrated Approach to measure the success of improvements and support

continuous improvement.

SUMMARY

In this chapter, the approach to the research design and the methodology through which the

purpose can be achieved is presented. First, the system boundaries of the Academies were

identified to elicit a rich picture of the system, including identifying stakeholders and the core

processes for achieving the system’s purpose, together with their unique measurement

parameters. From here, key outcomes of the system which were deemed to fulfil its purpose

formed criteria against which the evidence gathering process was defined. The data were collected

through questionnaires, interviews and from the Academies' own performance data. The values

were converted into the Perimeta Italian Flag model through a process which mapped confidence

against performance. All data from all cases were entered into the Perimeta model which was

based on the systems design. A visual report was then produced which provided evidence about

what was working well – what was working against desired outcomes and what was not known.

52

DATA ANALYSIS: SOCIAL SCIENCE

The data were analysed strand by strand using traditional social science analyses methods,

including descriptive statistics, t-tests and ANOVA as well as factor analyses. The findings from each

Strand are presented below.

STRAND ONE - DEVELOPMENT OF STUDENTS AS LEARNERS IN YEAR 7

Strand One was designed to assess Year 7 students’ self-reporting of their learning dispositions,

using the Effective Lifelong Learning Inventory's seven scales.

QUESTIONNAIRE AND FEEDBACK

The Effective Lifelong Learning Inventory (ELLI) questionnaire is an online survey consisting of seven

scales, totalling 72 items. These are research validated scales which measure the following student

learning dispositions: changing and learning, meaning making, creativity, critical curiosity, strategic

awareness, learning relationships and resilience.5 The items were structured with a four point likert

type scale labelled strongly disagree, disagree, agree, strongly agree. These were treated as

numerical scores at the stage of analysis in order to generate immediate feedback to students in

the form of a spider diagram (see Figure 8), feedback to teachers in the form of histograms and

mean scores and to provide raw data for analysis. The student feedback provided a framework for a

coaching conversation with the teacher or peers about learning. The histograms provided

diagnostic data for teachers about their class’s orientation to learning.

Table 17 Feedback to users in the form of a spider diagram representing the 7 dimensions of learning power

5 For reliability and validity statistics on these scales see Deakin Crick & Yu 2008, and Ren and Deakin Crick, 2013.

53

INTERVENTIONS

All of the Academies introduced changes in their approach to teaching and learning with the year

groups in this sample. These interventions were focused on coaching for learning power and

enquiry based learning.

THE SAMPLE

Students in Year 7 in each of the three Academies completed the Effective Lifelong Learning

Inventory (ELLI) between October and December of the academic year 2010/11. A sample of

teachers from each Academy were trained in the interpretation of the ELLI survey, coaching for

learning with students based on the data and the design of teaching and learning interventions to

stimulate students’ engagement in learning. At the end of the year the Academies completed a

post-test survey.

For technical and logistical reasons, from the total sample of 450 students, only 274 were robust

enough to contribute to data analysis, after careful screening and analysis. This was due to some

technical issues in retrieving data from a new provider and also because some of the Academies

had difficulty in students completing the post-tests within the time frame. Table 10 summarises the

sample for Strand 1.

Table 18 Strand One sample by Academy and gender

Sample by Gender and Academy

Academy Total

1 2 3

Gender Male 48 53 50 151

Female 66 24 33 123

Total 114 77 83 274

FINDINGS

In order to assess whether there was any difference between the first and second time of

administration of the ELLI, a paired samples t-test was computed with all complete data, combining

the three Academies. As can be seen from Tables 11 and 12, there were significant increases in

student scores on all seven learning power dimensions between Time 1 and Time 2, except for

Resilience where the mean score was significantly less in Time 2.

54

Table 19 Mean scores on each learning power scale Time 1 and Time 2

ELLI Scale Time one and Time Two Mean N Std. Deviation Std. Error Mean

Changing & Learning Time 1 68.64 139 20.27 1.71

Time 2 76.97 139 19.41 1.64

Learning Relationships Time 1 61.13 139 16.98 1.44

Time 2 69.34 139 16.46 1.39

Strategic Awareness Time 1 58.69 139 18.98 1.61

Time 2 68.10 139 18.88 1.60

Resilience Time 1 49.11 139 18.82 1.59

Time 2 44.73 139 18.95 1.60

Creativity Time 1 59.40 139 20.85 1.70

Time 2 69.32 139 17.91 1.51

Meaning Making Time 1 63.30 139 19.90 1.68

Time 2 74.75 139 18.84 1.59

Critical Curiosity Time 1 55.68 139 18.90 1.60

Time 2 71.16 139 18.48 1.56

Table 20 Paired t-tests

SD StE t df Sig(2 tailed)

Changing and Learning 18.98 1.61 -5.17 138 .000

Learning Relationships 17.24 1.46 -5.61 138 .000

Strategic Awareness 14.41 1.22 -7.69 138 .000

Resilience 16.97 1.43 3.04 138 .003

Creativity 18.01 1.52 -6.49 138 .000

Meaning Making 18.81 1.59 -7.17 138 .000

Critical Curiosity 12.54 1.06 -14.55 138 .000

DIFFERENCES BETWEEN ACADEMIES

In order to compare the differences between Academies an Analysis of Variance (ANOVA) was

computed at Time 1 and Time 2. This indicated that there were no significant differences between

the Academies at Time 1. However, the ANOVA computation for Time 2 shows significant

differences. Table 13 shows the descriptive statistics for each Academy at Time 2 against each

scale. Table 14 shows the ANOVA table with tests of significance for each scale also at Time 2. This

shows there to be significant differences between Academies in changing and learning, learning

relationships, strategic awareness and meaning making. However, there were no significant

differences in the other learning scales between Academies.

55

Table 21 Descriptive statistics for each Academy at Time 2

Table 22 ANOVA

between

Academies Time 2

ANOVA

Sum of df Mean Square F Sig.

N Mean Std. Deviation

CL.2: CL

Changing &

Learning

1 78 82.37 15.19

2 28 65.17 20.54

3 33 74.24 22.66

Total 139 76.97 19.41

LR.2: LR

Learning

Relationships

1 78 72.68 15.73

2 28 59.62 14.80

3 33 69.69 16.67

Total 139 69.34 16.46

SA.2: SA

Strategic Awareness

1 78 70.51 18.49

2 28 59.61 18.99

3 33 69.61 18.20

Total 139 68.10 18.88

RS.2: RS

Resilience

1 78 44.77 20.40

2 28 45.44 15.41

3 33 44.02 18.62

Total 139 44.73 18.95

CR.2: CR

Creativity

1 78 69.31 16.88

2 28 64.76 19.38

3 33 73.23 18.62

Total 139 69.32 17.91

MM.2: MM

Meaning Making

1 78 77.71 17.10

2 28 67.17 19.58

3 33 74.17 20.79

Total 139 74.75 18.84

CC.2: CC

Critical Curiosity

1 78 71.74 17.75

2 28 68.78 18.21

3 33 71.82 20.70

Total 139 71.18 18.48

56

Squares

CL.2: CL

Changing &

Learning

Between Groups 6414.550 2 3207.2 9.56 .000

Within Groups 45610.830 136 335.37

Total 52025.380 138

LR.2: LR

Learning

Relationships

Between Groups 3520.797 2 1760.39 7.06 .001

Within Groups 33877.507 136 249.09

Total 37398.304 138

SA.2: SA

Strategic

Awareness

Between Groups 2545.942 2 1272.97 3.70 .027

Within Groups 46673.191 136 343.18

Total 49219.133 138

RS.2: RS

Resilience

Between Groups 30.813 2 15.40 .04 .959

Within Groups 49554.488 136 364.37

Total 49585.300 138

CR.2: CR

Creativity

Between Groups 1086.832 2 543.41 1.71 .185

Within Groups 43206.053 136 317.69

Total 44292.886 138

MM.2: MM

Meaning

Making

Between Groups 2303.469 2 1151.73 3.35 .038

Within Groups 46705.813 136 343.42

Total 49009.282 138

CC.2: CC

Critical

Curiosity

Between Groups 199.881 2 99.94 .28 .749

Within Groups 46954.100 136 345.25

Total 47153.980 138

DIFFERENCES IN PRE-POST CHANGE BETWEEN ACADEMIES

In order to explore these differences further paired t-tests were conducted for the three Academies

separately. Although care should be taken in their interpretation as the sample size is not complete

and is different for each Academy, it appears that the contribution to the overall positive change

was not evenly distributed between the Academies.

Academy 1

Table 23 Descriptive Statistics Academy 1

Mean N Std. Deviation Std. Error Mean

Pair 1 CL.1: CL 71.26 78 19.86 2.25

CL.2: CL 82.37 78 15.19 1.72

Pair 2 LR.1: LR 64.67 78 15.11 1.71

LR.2: LR 72.69 78 15.74 1.78

Pair 3 SA.1: SA 61.54 78 18.38 2.08

SA.2: SA 70.51 78 18.49 2.09

Pair 4 RS.1: RS 48.57 78 19.36 2.19

57

RS.2: RS 44.77 78 20.40 2.31

Pair 5 CR.1: CR 60.90 78 19.46 2.20

CR.2: CR 69.32 78 16.89 1.91

Pair 6 MM.1: MM 66.00 78 18.89 2.14

MM.2: MM 77.72 78 17.10 1.94

Pair 7 CC.1: CC 57.88 78 18.05 2.04

CC.2: CC 71.75 78 17.76 2.01

Table 24 Paired t-test Academy 1

Mean Std. Deviation

Std. Error Mean

t df Sig. (2-tailed)

Pair 1 CL.1: CL - CL.2: CL -11.11

18.54 2.10 -5.29 77 0.00

Pair 2 LR.1: LR - LR.2: LR -8.01 15.72 1.78 -4.50 77 0.00

Pair 3 SA.1: SA - SA.2: SA -8.97 12.99 1.47 -6.10 77 0.00

Pair 4 RS.1: RS - RS.2: RS 3.80 17.13 1.94 1.96 77 0.05

Pair 5 CR.1: CR - CR.2: CR -8.42 16.58 1.88 -4.48 77 0.00

Pair 6 MM.1: MM - MM.2: MM -11.72

17.85 2.02 -5.80 77 0.00

Pair 7 CC.1: CC - CC.2: CC -13.87

10.43 1.18 -11.74 77 0.00

Academy 2

Table 25 Descriptive Statistics Academy 2

Paired Samples Statistics

Mean N Std. Deviation Std. Error Mean

Pair 1 CL.1: CL 59.52 28 22.07 4.17

CL.2: CL 65.18 28 20.54 3.88

Pair 2 LR.1: LR 54.17 28 17.12 3.23

LR.2: LR 59.62 28 14.80 2.80

Pair 3 SA.1: SA 49.45 28 15.87 3.00

SA.2: SA 59.62 28 18.99 3.59

Pair 4 RS.1: RS 49.86 28 19.23 3.63

RS.2: RS 45.45 28 15.41 2.91

Pair 5 CR.1: CR 53.33 28 22.88 4.32

CR.2: CR 64.76 28 19.38 3.66

Pair 6 MM.1: MM 55.10 28 21.23 4.01

MM.2: MM 67.18 28 19.58 3.70

Pair 7 CC.1: CC 50.13 28 18.98 3.59

CC.2: CC 68.78 28 18.21 3.44

58

Table 26 Paired t-tests Academy 2

Mean Std.

Deviation

Std. Error

Mean

t df Sig. (2-

tailed)

Pair 1 CL.1: CL - CL.2: CL -5.65 21.28 4.02 -1.41 27 0.17

Pair 2 LR.1: LR - LR.2: LR -5.46 17.72 3.35 -1.63 27 0.12

Pair 3 SA.1: SA - SA.2: SA -10.16 17.70 3.35 -3.04 27 0.01

Pair 4 RS.1: RS - RS.2: RS 4.41 18.65 3.53 1.25 27 0.22

Pair 5 CR.1: CR - CR.2: CR -11.43 22.54 4.26 -2.68 27 0.01

Pair 6 MM.1: MM - MM.2: MM

-12.07 22.31 4.22 -2.86 27 0.01

Pair 7 CC.1: CC - CC.2: CC -18.65 16.08 3.04 -6.14 27 0.00

Academy 3

Table 27 Descriptive Statistics Academy 3

Mean N Std.

Deviation

Std. Error

Mean

Pair 1 CL.1: CL 70.20 33 17.93 3.12

CL.2: CL 74.24 33 22.67 3.95

Pair 2 LR.1: LR 58.67 33 19.18 3.34

LR.2: LR 69.70 33 16.67 2.90

Pair 3 SA.1: SA 59.83 33 20.82 3.62

SA.2: SA 69.62 33 18.20 3.17

Pair 4 RS.1: RS 49.79 33 17.67 3.08

RS.2: RS 44.03 33 18.62 3.24

Pair 5 CR.1: CR 61.01 33 21.96 3.82

CR.2: CR 73.23 33 18.62 3.24

Pair 6 MM.1: MM 63.93 33 19.81 3.45

MM.2: MM 74.17 33 20.79 3.62

Pair 7 CC.1: CC 55.22 33 20.38 3.55

CC.2: CC 71.83 33 20.70 3.60

Table 28 Paired t-test Academy 3

Mean Std.

Deviation

Std. Error

Mean

t df Sig. (2-tailed)

CL.1: CL - CL.2: CL -4.04 17.32 3.02 -1.34 32 0.19

LR.1: LR - LR.2: LR -11.03 20.20 3.52 -3.14 32 0.00

SA.1: SA - SA.2: SA -9.79 14.97 2.61 -3.76 32 0.00

RS.1: RS - RS.2: RS 5.76 15.49 2.70 2.14 32 0.04

CR.1: CR - CR.2: CR -12.22 17.21 3.00 -4.08 32 0.00

MM.1: MM - MM.2: MM -10.25 18.37 3.20 -3.20 32 0.00

CC.1: CC - CC.2: CC -16.61 13.49 2.35 -7.08 32 0.00

59

SUMMARY AND FINDINGS FROM STRAND ONE

Strand One was designed to assess Year 7 students’ self-reporting of their learning dispositions,

using the Effective Lifelong Learning Inventory's seven scales.

The findings demonstrated there was a significant increase in mean score between Time 1 and Time

2 on six of the seven dimensions of learning power, based on the Effective Lifelong Learning

Inventory, which means that something other than chance must account for this increase. The

increase was not uniform across all three Academies where Academy 1 and 3 were more similar in

terms of the statistical significance in many of the dimensions of learning power.

This seems to suggest that whilst there are significant increases in scores for all the Academies

overall (except for Resilience), there are other factors which contributed to the differing levels of

significance in the seven dimensions between Academies. This could be due to a range of factors

such as the students themselves, the delivery of the interventions or the overall leadership of the

Academy and are worth exploring further.

STRAND TWO - STUDENT PERCEPTIONS OF THE IMPACT OF THE EDUCATION CHARTER

Strand Two was a questionnaire designed to assess students' perceptions about their learning,

teaching, progress and achievement in an Oasis Academy.

QUESTIONNAIRE

The Learning of Students questionnaire consisted of 42 items structured with a four-point Likert-

scale labeled, strongly disagree, disagree, agree, and strongly agree. The total number of student

respondents was 377. The students were selected from Years 8 and 10.

Table 29 Data Description for Strand Two

Variable Defined value

Missing

Academy 1, 3

Gender 1, 2 0.3% (1 record)

S21 – S242

1-4 0.0-4.0% (some records with value=0 which are treated as missing leading to 322 records without any missing on these variables)

STUDENT RESPONSES BY QUESTION

The following Table 22 shows students' responses by question, with the number of students

responding to each question and the percentage of the total.

Table 30 Strand 2 Student questionnaire responses by item

60

Item 1 strongly

disagree

2 disagree 3 agree 4 strongly

agree

missing

N % N % N % N % Total %

S21 1. I feel included in the activities that are available to me in the Academy

27 7.2 75 19.9 230 61.0 45 11.9 377 0.0

S22 2. I feel safe in the Academy 22 5.8 43 11.4 204 54.1 108 28.6 377 0.0

S23 3. I feel valued and cared for as an individual in the Academy

32 8.5 84 22.3 204 54.1 54 14.3 374 0.8

S24 4. I like learning new things 8 2.1 37 9.8 203 53.8 128 34.0 376 0.3

S25 5. I usually concentrate on my learning

9 2.4 44 11.7 224 59.4 98 26.0 375 0.5

S26 6. I am hard to distract 74 19.6 184 48.8 91 24.1 27 7.2 376 0.3

S27 7. I like working with other students to help my learning

20 5.3 63 16.7 180 47.7 111 29.4 374 0.8

S28 9. I am confident in my learning 40 10.6 96 25.5 167 44.3 74 19.6 377 0.0

S29 8. I don't distract other students 19 5.0 49 13.0 206 54.6 102 27.1 376 0.3

S210 10. I take a lot of care with my work 8 2.1 46 12.2 233 61.8 90 23.9 377 0.0

S211 11. I work hard 9 2.4 39 10.3 217 57.6 111 29.4 376 0.3

S212 12. I do more than I am asked to do in class

31 8.2 156 41.4 155 41.1 33 8.8 375 0.5

S213 13. I don't mind making mistakes because I learn from them

23 6.1 84 22.3 182 48.3 87 23.1 376 0.3

S214 14. I come up with new ideas to help my learning

23 6.1 102 27.1 189 50.1 62 16.4 376 0.3

S215 15. I enjoy my learning 32 8.5 87 23.1 194 51.5 63 16.7 376 0.3

S216 16. My teachers teach well so that I learn successfully

27 7.2 78 20.7 198 52.5 70 18.6 373 1.1

S217 17. I have regular opportunities to express my opinion to my teachers about my learning

42 11.1 126 33.4 158 41.9 45 11.9 371 1.6

S218 18. I often have conversations with my teachers which help me to make better progress

52 13.8 140 37.1 138 36.6 42 11.1 372 1.3

S219 19. My teachers seem to enjoy teaching my classes

36 9.5 116 30.8 176 46.7 46 12.2 374 0.8

S220 20. My teachers often share things that they have learned themselves

25 6.6 83 22.0 202 53.6 63 16.7 373 1.1

S221 21. I get on well with my teachers 21 5.6 71 18.8 191 50.7 91 24.1 374 0.8

S222 22. I get extra support to help my learning when I need it

60 15.9 95 25.2 173 45.9 46 12.2 374 0.8

S223 23. I know that the senior teachers want us to learn successfully

17 4.5 53 14.1 214 56.8 90 23.9 374 0.8

S224 24. Everyone works very hard to make the Academy a great place to learn and to do well

23 6.1 101 26.8 176 46.7 74 19.6 374 0.8

S225 25. My teachers constantly expect me to improve on my personal best

9 2.4 53 14.1 184 48.8 128 34.0 374 0.8

S226 26. Right now I am achieving the best I possibly can

23 6.1 75 19.9 178 47.2 93 24.7 369 2.1

61

S227 27. From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the Academy

16 4.2 56 14.9 189 50.1 108 28.6 369 2.1

S228 28. I have kept up a good rate of progress ever since my first year at the Academy

14 3.7 73 19.4 190 50.4 90 23.9 367 2.7

S229 29. I think that my results at the moment are as good as I can do

25 6.6 117 31.0 153 40.6 72 19.1 367 2.7

S230 30. I feel that my successes at the Academy are recognised

43 11.4 115 30.5 172 45.6 38 10.1 368 2.4

S231 31. I feel that my successes with activities outside the Academy are recognised

70 18.6 112 29.7 142 37.7 43 11.4 367 2.7

S232 32. I feel proud about what I have achieved so far

14 3.7 82 21.8 176 46.7 96 25.5 368 2.4

S233 33. We are taught about the responsibilities of students as well as their rights

21 5.6 75 19.9 193 51.2 76 20.2 365 3.2

S234 34. We care about helping each other in the Academy community

28 7.4 83 22.0 195 51.7 61 16.2 367 2.7

S235 35. We are encouraged to care about the needs of other people in the local community and around the world

17 4.5 80 21.2 193 51.2 77 20.4 367 2.7

S236 36. I regularly offer to help others 25 6.6 89 23.6 179 47.5 75 19.9 368 2.4

S237 37. My parents or carers feel that they are always welcome in the Academy

27 7.2 56 14.9 179 47.5 100 26.5 362 4.0

S238 38. My parents or carers feel involved with my learning and my progress

28 7.4 63 16.7 180 47.7 95 25.2 366 2.9

S239 39. My parents or carers are pleased with my progress

21 5.6 50 13.3 181 48.0 113 30.0 365 3.2

S240 40. My parents or carers have always been able to sort out any problems that I have had

27 7.2 58 15.4 154 40.8 127 33.7 366 2.9

S241 41. I am encouraged to lead activities in the classroom

57 15.1 114 30.2 156 41.4 42 11.1 369 2.1

S242 42. I have had opportunities to lead other activities at the Academy

85 22.5 102 27.1 125 33.2 57 15.1 369 2.1

DIFFERENCES BETWEEN SCHOOLS BY QUESTIONS

Individually, independent t-tests were carried out on each item to examine whether there were

statistically significant mean differences in individual items between the Academies.

It was found that, on average, there were no mean differences in agreement or disagreement

between Academy 1 and Academy 3 student respondents to 10 out of 42 items. These include the

statements about ‘feel safe’ in the school (S22), ‘like work with other students’ (S27)or ‘take a lot of

care’ (S210) to help their learning or work, the expectation from teachers to improve on ‘personal

best’ (S225), achieving their best as they ‘possibly can’ (S226), ‘feel proud’ of what they have

62

achieved, their results ‘are as good as’ they can do (S229), parents or carers pleased with their

‘learning and progress’ (S232), they being encouraged to ‘lead activities in the classroom (S239),

and being offered opportunities to ‘lead other activities’ in the school (S241) (see shaded areas in

the Table 23 below).

However, Academy 1 student respondents reported averagely higher mean responses than their

counterparts of Academy 3 to the remaining 32 items at α = 0.05 level of statistical significance (see

non-shaded areas in the Table 26. below). In other words, to some degree, students in Academy 1

tend to agree more7 than Academy 3 students to the statements of, for example, ‘like learning new

things’ (S24),’concentrate on my learning (S25), ‘work hard’ (SS211), not minding ‘making mistakes’

(S213), ‘get on well’ with their teachers (S221), knowing that ‘the senior teachers want us to

learning successfully’ (S223), keeping up ‘a good rate of progress’ (S227), and their parents or carers

being involved with their learning and progress (S238) as well as being able to ‘sort out any

problems’ that they have had (S240). On the other hand, Academy 1 student respondents are also

likely to disagree less8 than Academy 3 student respondents on ‘feel valued and cared’ (S23), being

‘confident in my learning’ (S28), their teachers ‘teach well’ (S216), ‘everyone works very hard to

make’ their school a great place to learn (S224), and being ‘taught about the responsibilities of

students as well as their rights’ (S233).

Table 31 t-test for Equality of Means for each item (purple shaded = non significance at α = 0.05)

t-test for Equality of Means

(purple shaded = non significance

at α = 0.05)

t df Sig.

(2-

taile

d)

Mean Diff.

Ac Mn N SD Total Mean N SD

S21 1. I feel included in the activities that are available to me in the Academy

5.94 375.00 0.00 0.46 1 2.94 244 0.61

5.36 203.08 0.00 0.46 3 2.48 133 0.88 Total 2.78 377 0.75

S22 2. I feel safe in the Academy

1.96 375.00 0.05 0.17 1 3.11 244 0.66

1.75 198.36 0.08 0.17 3 2.95 133 0.99 Total 3.06 377 0.80

S23 3. I feel valued and cared for as an individual in the Academy

3.28 372.00 0.00 0.28 1 2.85 243 0.64

2.87 186.27 0.00 0.28 3 2.56 131 1.02 Total 2.75 374 0.81

S24 4. I like learning new things

3.68 374.00 0.00 0.27 1 3.30 244 0.55

3.21 186.31 0.00 0.27 3 3.02 132 0.89 Total 3.20 376 0.70

S25 5. I usually concentrate on my learning

2.82 373.00 0.01 0.21 1 3.17 243 0.54

2.47 185.83 0.01 0.21 3 2.96 132 0.88 Total 3.10 375 0.68

S26 6. I am hard to distract

2.87 374.00 0.00 0.26 1 2.28 244 0.73

2.64 211.47 0.01 0.26 3 2.02 132 0.98 Total 2.19 376 0.83

S27 7. I like working with other students to help my

1.55 372.00 0.12 0.14 1 3.07 243 0.72

1.41 204.20 0.16 0.14 3 2.93 131 0.99 Total 3.02 374 0.82

7 Referring to Academy 1 students reporting a mean of greater than 3.00 to the exampled statements

8 Referring to Academy 1 students reporting a mean of equal to or less than 3.00 to the exampled statements

63

learning

S28 9. I am confident in my learning

5.49 375.00 0.00 0.51 1 2.91 244 0.71

4.88 195.62 0.00 0.51 3 2.40 133 1.09 Total 2.73 377 0.90

S29 8. I don't distract other students

2.84 374.00 0.00 0.24 1 3.12 244 0.70

2.65 219.91 0.01 0.24 3 2.89 132 0.89 Total 3.04 376 0.78

S210 10. I take a lot of care with my work

1.28 375.00 0.20 0.09 1 3.11 244 0.59

1.18 216.73 0.24 0.09 3 3.02 133 0.78 Total 3.07 377 0.66

S211 11. I work hard

3.00 374.00 0.00 0.22 1 3.22 244 0.57

2.69 197.58 0.01 0.22 3 3.00 132 0.85 Total 3.14 376 0.69

S212 12. I do more than I am asked to do in class

4.82 373.00 0.00 0.39 1 2.64 242 0.66

4.41 211.47 0.00 0.39 3 2.26 133 0.89 Total 2.51 375 0.77

S213 13. I don't mind making mistakes because I learn from them

4.50 374.00 0.00 0.39 1 3.02 243 0.69

4.06 202.00 0.00 0.39 3 2.63 133 1.00 Total 2.89 376 0.83

S214 14. I come up with new ideas to help my learning

4.39 374.00 0.00 0.37 1 2.90 243 0.67

4.00 208.55 0.00 0.37 3 2.53 133 0.93 Total 2.77 376 0.79

S215 15. I enjoy my learning

6.42 374.00 0.00 0.55 1 2.96 243 0.68

5.83 206.62 0.00 0.55 3 2.41 133 0.95 Total 2.77 376 0.83

S216 16. My teachers teach well so that I learn successfully

4.01 371.00 0.00 0.35 1 2.95 243 0.69

3.63 200.25 0.00 0.35 3 2.61 130 0.97 Total 2.83 373 0.81

S217 17. I have regular opportunities to express my opinion to my teachers about my learning

3.82 369.00 0.00 0.35 1 2.68 241 0.76

3.57 219.21 0.00 0.35 3 2.33 130 0.95 Total 2.56 371 0.85

S218 18. I often have conversations with my teachers which help me to make better progress

3.69 370.00 0.00 0.34 1 2.58 243 0.75

3.37 203.91 0.00 0.34 3 2.23 129 1.02 Total 2.46 372 0.87

S219 19. My teachers seem to enjoy teaching my classes

5.28 372.00 0.00 0.46 1 2.78 244 0.67

4.71 192.92 0.00 0.46 3 2.32 130 0.99 Total 2.62 374 0.82

S220 20. My teachers often share things that they have learned themselves

4.27 371.00 0.00 0.36 1 2.94 242 0.67

3.87 203.58 0.00 0.36 3 2.58 131 0.94 Total 2.81 373 0.79

S221 21. I get on well with my teachers

3.15 372.00 0.00 0.27 1 3.04 243 0.69

2.86 203.79 0.00 0.27 3 2.76 131 0.97 Total 2.94 374 0.81

S222 22. I get extra support to help my learning when I need i

6.41 372.00 0.00 0.60 1 2.76 243 0.76

5.87 208.56 0.00 0.60 3 2.16 131 1.02 Total 2.55 374 0.90

S223 23. I know that the senior teachers want us to learn successfully

3.98 372.00 0.00 0.32 1 3.12 243 0.62

3.56 196.23 0.00 0.32 3 2.80 131 0.92 Total 3.01 374 0.75

S224 24. Everyone works very hard to make the Academy a great place to learn and to do well

3.29 372.00 0.00 0.29 1 2.91 244 0.76

3.11 225.17 0.00 0.29 3 2.62 130 0.91 Total 2.80 374 0.82

S225 25. My teachers constantly expect me to improve on my personal best

1.57 372.00 0.12 0.13 1 3.20 244 0.66

1.45 208.13 0.15 0.13 3 3.07 130 0.88 Total 3.15 374 0.75

S226 26. Right no w I am achieving the best I possibly can

1.19 367.00 0.23 0.11 1 2.96 239 0.74

1.09 206.80 0.28 0.11 3 2.85 130 1.00 Total 2.92 369 0.84

S227 27. From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the Academy

3.53 367.00 0.00 0.30 1 3.16 239 0.70

3.28 215.54 0.00 0.30 3 2.86 130 0.90 Total 3.05 369 0.79

64

S228 28. I have kept up a good rate of progress ever since my first year at the Academy

3.46 365.00 0.00 0.29 1 3.07 238 0.66

3.15 201.58 0.00 0.29 3 2.78 129 0.92 Total 2.97 367 0.77

S229 29. I think that my results at the moment are as good as I can do

-0.95 365.00 0.34 -0.09 1 2.71 238 0.83

-0.93 247.91 0.35 -0.09 3 2.80 129 0.89 Total 2.74 367 0.85

S230 30. I feel that my successes at the Academy are recognised

2.84 366.00 0.00 0.25 1 2.65 238 0.71

2.57 200.84 0.01 0.25 3 2.39 130 1.00 Total 2.56 368 0.83

S231 31. I feel that my successes with activities outside the Academy are recognised

2.31 365.00 0.02 0.23 1 2.51 238 0.78

2.08 195.17 0.04 0.23 3 2.28 129 1.14 Total 2.43 367 0.93

S232 32. I feel proud about what I have achieved so far

1.11 366.00 0.27 0.10 1 3.00 239 0.70

1.01 204.70 0.31 0.10 3 2.90 129 0.95 Total 2.96 368 0.80

S233 33. We are taught about the responsibilities of students as well as their rights

3.86 363.00 0.00 0.33 1 3.00 236 0.68

3.52 203.40 0.00 0.33 3 2.67 129 0.94 Total 2.89 365 0.80

S234 34. We care about helping each other in the Academy community

4.05 365.00 0.00 0.35 1 2.91 237 0.65

3.60 191.48 0.00 0.35 3 2.56 130 1.00 Total 2.79 367 0.81

S235 35. We are encouraged to care about the needs of other people in the local community and around the world

2.41 365.00 0.02 0.20 1 2.97 238 0.68

2.20 204.29 0.03 0.20 3 2.77 129 0.92 Total 2.90 367 0.78

S236 36. I regularly offer to help others

4.50 366.00 0.00 0.40 1 2.97 238 0.72

4.14 209.62 0.00 0.40 3 2.57 130 0.96 Total 2.83 368 0.83

S237 37. My parents or carers feel that they are always welcome in the Academy

3.86 360.00 0.00 0.36 1 3.10 234 0.71

3.48 195.29 0.00 0.36 3 2.74 128 1.03 Total 2.97 362 0.85

S238 38. My parents or carers feel involved with my learning and my progress

5.22 364.00 0.00 0.47 1 3.10 237 0.72

4.75 201.18 0.00 0.47 3 2.63 129 1.00 Total 2.93 366 0.86

S239 39. My parents or carers are pleased with my progress

1.93 363.00 0.05 0.17 1 3.12 236 0.71

1.76 202.03 0.08 0.17 3 2.95 129 0.99 Total 3.06 365 0.82

S240 40. My parents or carers have always been able to sort out any problems that I have had

3.90 364.00 0.00 0.37 1 3.17 237 0.75

3.51 196.48 0.00 0.37 3 2.80 129 1.08 Total 3.04 366 0.90

S241 41. I am encouraged to lead activities in the classroom

1.97 367.00 0.05 0.19 1 2.56 240 0.77

1.79 200.72 0.07 0.19 3 2.37 129 1.07 Total 2.50 369 0.89

S242 42. I have had opportunities to lead other activities at the Academy

0.42 367.00 0.68 0.05 1 2.43 240 0.92

0.39 217.03 0.70 0.05 3 2.39 129 1.15 Total 2.42 369 1.01

GENDER DIFFERENCE IN ITEM MEAN

Individual independent t-tests were carried out on each item to examine whether there were

statistically significant mean differences between genders.

65

The statistical results indicate that male and female student respondents on average provided

similar responses to 20 out of 42 items such as ‘feel included’ in available activities (S221), ‘like

learning new things’ (S224), ‘don’t distract other students’ (S229), ‘feel’ their successes at the

Academy recognised (S231), also about their teachers ‘teach well’ (S216) and ‘seem to enjoy

teaching’ their classes (S219) as well as ‘expect me to improve on my personal best’ (S225), in

addition to the statements about their parents or carers feeling that they are ‘always welcome in

the Academy’ (SS237), ‘involved with my learning and my progress (S238), and ‘pleased with my

progress’ (S240) (see shaded areas in the Table 24 below).

Interestingly, across the Academies female student respondents reported higher means than male

student respondents on the remaining 22 items at α = 0.05 level of statistical significance. For

example, female students generally agree more 9 than male counterparts that they ‘feel safe in the

Academy’ (S22), ‘concentrate on my learning’ (S25), ‘work with other students’ (S27), ‘care with my

work’ (S210) and ‘work hard’ (SS211), ‘get on well with my teachers’ (S221), keeping up ‘a good rate

of progress’ (S227) and ‘feel proud’ about what they have achieved (S232). And female students

also on average disagree less10 than male students on ‘confident in my learning’ (S28), ‘do more’

than being asked to do in the class (S212), ‘enjoying my learning’ (S215), ‘get extra support’ to help

their learning (S222), and their results ‘are as good as’ they can do (S229).

Table 32t-test for Equality of Means for each item by gender (purple shaded = non significance at α = 0.05)

Item t df Sig. (2-

tailed)

Mean

Diff.

Gender N Mean SD

S21 1. I feel included in the activities that are available to me in the Academy

-0.94 374.00 0.35 -0.07 1 Male 198 2.74 0.82

-0.95 369.67 0.34 -0.07 2 Female 178 2.81 0.66

S22 2. I feel safe in the Academy -2.62 374.00 0.01 -0.21 1 Male 198 2.95 0.87

-2.66 367.22 0.01 -0.21 2 Female 178 3.17 0.68

S23 3. I feel valued and cared for as an individual in the Academy

-0.37 371.00 0.71 -0.03 1 Male 197 2.74 0.82

-0.37 367.90 0.71 -0.03 2 Female 176 2.77 0.80

S24 4. I like learning new things 0.24 373.00 0.81 0.02 1 Male 197 3.21 0.72

0.24 372.75 0.81 0.02 2 Female 178 3.19 0.67

S25 5. I usually concentrate on my learning -3.80 372.00 0.00 -0.26 1 Male 198

2.97 0.69

-3.82 371.43 0.00 -0.26 2 Female 176 3.24 0.64

S26 6. I am hard to distract -1.68 373.00 0.09 -0.14 1 Male 198 2.12 0.85

-1.69 372.10 0.09 -0.14 2 Female 177 2.26 0.80

S27 7. I like working with other students to help my learning

-2.11 371.00 0.04 -0.18 1 Male 196 2.94 0.83

-2.12 368.25 0.04 -0.18 2 Female 177 3.12 0.81

S28 9. I am confident in my learning -3.78 374.00 0.00 -0.34 1 Male 198 2.57 0.94

-3.81 373.72 0.00 -0.34 2 Female 178 2.91 0.82

S29 8. I don't distract other students -1.05 373.00 0.29 -0.08 1 Male 198 3.00 0.80

-1.06 372.00 0.29 -0.08 2 Female 177 3.08 0.75

S210 10. I take a lot of care with my work -4.92 374.00 0.00 -0.33 1 Male 198 2.92 0.71

-4.98 366.93 0.00 -0.33 2 Female 178 3.25 0.56

9 Referring to female students reporting a mean of greater than 3.00 to the exampled statements

10 Referring to female students reporting a mean of equal to or less than 3.00 to the exampled statements

66

S211 11. I work hard -3.24 373.00 0.00 -0.23 1 Male 197 3.04 0.77

-3.29 361.36 0.00 -0.23 2 Female 178 3.26 0.58

S212 12. I do more than I am asked to do in class -3.07 372.00 0.00 -0.24 1 Male 196 2.39 0.79

-3.08 371.98 0.00 -0.24 2 Female 178 2.63 0.73

S213 13. I don't mind making mistakes because I learn from them

-2.04 373.00 0.04 -0.17 1 Male 198 2.80 0.84

-2.04 370.81 0.04 -0.17 2 Female 177 2.98 0.81

S214 14. I come up with new ideas to help my learning

-0.28 373.00 0.78 -0.02 1 Male 198 2.76 0.79

-0.28 367.89 0.78 -0.02 2 Female 177 2.79 0.80

S215 15. I enjoy my learning -2.68 373.00 0.01 -0.23 1 Male 197 2.66 0.87

-2.70 372.76 0.01 -0.23 2 Female 178 2.89 0.77

S216 16. My teachers teach well so that I learn successfully

-0.96 370.00 0.34 -0.08 1 Male 195 2.79 0.86

-0.96 369.68 0.34 -0.08 2 Female 177 2.88 0.76

S217 17. I have regular opportunities to express my opinion to my teachers about my learning

-1.78 368.00 0.08 -0.16 1 Male 194 2.48 0.89

-1.79 367.94 0.07 -0.16 2 Female 176 2.64 0.80

S218 18. I often have conversations with my teachers which help me to make better progress

-1.18 369.00 0.24 -0.11 1 Male 195 2.41 0.92

-1.18 368.91 0.24 -0.11 2 Female 176 2.51 0.81

S219 19. My teachers seem to enjoy teaching my classes

-1.81 371.00 0.07 -0.15 1 Male 195 2.55 0.83

-1.81 368.82 0.07 -0.15 2 Female 178 2.70 0.81

S220 20. My teachers often share things that they have learned themselves

-0.80 370.00 0.42 -0.07 1 Male 196 2.78 0.83

-0.81 369.95 0.42 -0.07 2 Female 176 2.85 0.75

S221 21. I get on well with my teachers -3.11 371.00 0.00 -0.26 1 Male 195 2.82 0.85

-3.13 370.63 0.00 -0.26 2 Female 178 3.08 0.75

S222 22. I get extra support to help my learning when I need it

-3.22 371.00 0.00 -0.30 1 Male 196 2.41 0.93

-3.23 370.97 0.00 -0.30 2 Female 177 2.71 0.85

S223 23. I know that the senior teachers want us to learn successfully

-1.46 371.00 0.15 -0.11 1 Male 195 2.95 0.83

-1.47 362.69 0.14 -0.11 2 Female 178 3.07 0.65

S224 24. Everyone works very hard to make the Academy a great place to learn and to do well

-2.90 371.00 0.00 -0.25 1 Male 195 2.69 0.87

-2.92 370.23 0.00 -0.25 2 Female 178 2.93 0.76

S225 25. My teachers constantly expect me to improve on my personal best

-1.92 371.00 0.06 -0.15 1 Male 195 3.08 0.78

-1.93 370.92 0.05 -0.15 2 Female 178 3.23 0.70

S226 26. Right now I am achieving the best I possibly can

-3.85 366.00 0.00 -0.33 1 Male 193 2.77 0.84

-3.85 364.61 0.00 -0.33 2 Female 175 3.10 0.81

S227 27. From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the Academy

-2.41 366.00 0.02 -0.20 1 Male 193 2.96 0.83

-2.43 365.11 0.02 -0.20 2 Female 175 3.16 0.72

S228 28. I have kept up a good rate of progress ever since my first year at the Academy

-2.34 364.00 0.02 -0.19 1 Male 192 2.88 0.82

-2.36 363.28 0.02 -0.19 2 Female 174 3.07 0.71

S229 29. I think that my results at the moment are as good as I can do

-2.69 364.00 0.01 -0.24 1 Male 192 2.63 0.84

-2.69 359.92 0.01 -0.24 2 Female 174 2.87 0.85

S230 30. I feel that my successes at the Academy are recognised

-1.60 365.00 0.11 -0.14 1 Male 192 2.49 0.89

-1.62 362.96 0.11 -0.14 2 Female 175 2.63 0.75

S231 31. I feel that my successes with activities outside the Academy are recognised

-1.39 364.00 0.16 -0.14 1 Male 192 2.36 0.99

-1.40 363.37 0.16 -0.14 2 Female 174 2.50 0.86

67

S232 32. I feel proud about what I have achieved so far

-3.40 365.00 0.00 -0.28 1 Male 193 2.83 0.85

-3.43 363.65 0.00 -0.28 2 Female 174 3.11 0.72

S233 33. We are taught about the responsibilities of students as well as their rights

-2.12 362.00 0.03 -0.18 1 Male 191 2.81 0.84

-2.14 361.60 0.03 -0.18 2 Female 173 2.98 0.74

S234 34. We care about helping each other in the Academy community

-4.27 364.00 0.00 -0.35 1 Male 191 2.62 0.89

-4.32 349.81 0.00 -0.35 2 Female 175 2.97 0.66

S235 35. We are encouraged to care about the needs of other people in the local community and around the world

-3.30 364.00 0.00 -0.27 1 Male 191 2.77 0.79

-3.31 363.90 0.00 -0.27 2 Female 175 3.04 0.74

S236 36. I regularly offer to help others -3.12 365.00 0.00 -0.27 1 Male 192 2.70 0.88

-3.14 363.28 0.00 -0.27 2 Female 175 2.97 0.75

S237 37. My parents or carers feel that they are always welcome in the Academy

-1.58 359.00 0.12 -0.14 1 Male 188 2.90 0.88

-1.58 358.98 0.11 -0.14 2 Female 173 3.05 0.82

S238 38. My parents or carers feel involved with my learning and my progress

-1.15 363.00 0.25 -0.10 1 Male 191 2.88 0.89

-1.16 362.98 0.25 -0.10 2 Female 174 2.99 0.82

S239 39. My parents or carers are pleased with my progress

-1.84 362.00 0.07 -0.16 1 Male 189 2.98 0.86

-1.85 361.83 0.07 -0.16 2 Female 175 3.14 0.78

S240 40. My parents or carers have always been able to sort out any problems that I have had

-1.44 363.00 0.15 -0.13 1 Male 190 2.97 0.92

-1.44 362.73 0.15 -0.13 2 Female 175 3.11 0.87

S241 41. I am encouraged to lead activities in the classroom

-1.53 366.00 0.13 -0.14 1 Male 191 2.43 0.87

-1.52 361.66 0.13 -0.14 2 Female 177 2.57 0.90

S242 42. I have had opportunities to lead other activities at the Academy

-0.77 366.00 0.44 -0.08 1 Male 191 2.38 1.07

-0.77 365.08 0.44 -0.08 2 Female 177 2.46 0.94

UNDERLYING THEMES IN THE DATA

An exploratory factor analysis was conducted to investigate whether there are any potential

themes underlying the items. The following rotated component matrix deleted six items (S21, S22,

S26, S27, S28, S211, S214) and produced the following six themes or factors. These factors could

form the basis for a more powerful scale for future surveys and reduction of data in a future

Perimeta model. They require further interpretation, however they represent key themes through

which the students understood or construed their experience at their Academy.

Table 33 Underlying themes: rotated component matrix from exploratory factor analysis with alpha reliability co-efficient for each resulting scale

1 2 3 4 5 6 α S222 22. I get extra support to help my learning when I need it .645 .203 .023 .125 .119 .110

S217 17. I have regular opportunities to express my opinion to my teachers about my learning

.639 .196 .173 .105 .036 .277

S216 16. My teachers teach well so that I learn successfully .634 .239 .285 .253 .017 -.062 S23 3. I feel valued and cared for as an individual in the Academy .573 -.066 .352 .229 .202 .105

S218 18. I often have conversations with my teachers which help me to make better progress

.565 .184 .023 .161 .072 .379

S233 33. We are taught about the responsibilities of students as well as their rights

.510 .112 .087 .375 .289 .141

68

S230 30. I feel that my successes at the Academy are recognised .474 -.040 .349 .021 .263 .391 S219 19. My teachers seem to enjoy teaching my classes .427 .140 .232 .412 .130 .094

S220 20. My teachers often share things that they have learned themselves

.411 .237 .020 .216 .268 -.017 0.84

S24 4. I like learning new things .112 .661 .078 -.063 .146 -.041 S215 15. I enjoy my learning .294 .646 .155 .104 .285 .030

S25 5. I usually concentrate on my learning .085 .607 .329 .293 .097 .058 S236 36. I regularly offer to help others .104 .530 .130 .132 .273 .244 S29 8. I don't distract other students .167 .527 .359 .085 .145 .079

S210 10. I take a lot of care with my work -.004 .517 .428 .267 -.036 .139 S213 13. I don't mind making mistakes because I learn from them .222 .514 .026 .308 .129 -.007

S212 12. I do more than I am asked to do in class .165 .501 .235 .134 -.077 .433 0.82 S226 26. Right now I am achieving the best I possibly can .058 .222 .703 .229 .124 .166

S228 28. I have kept up a good rate of progress ever since my first year at the Academy

.236 .289 .655 .046 .221 .032

S227 27. From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the

Academy

.313 .184 .654 .019 .231 -.138

S229 29. I think that my results at the moment are as good as I can do .110 .106 .585 .149 .100 .288 S232 32. I feel proud about what I have achieved so far .112 .220 .539 .317 .248 .219 0.81

S224 24. Everyone works very hard to make the Academy a great place to learn and to do well

.238 .048 .064 .685 .104 .220

S234 34. We care about helping each other in the Academy community

.278 .106 .117 .609 .262 .254

S235 35. We are encouraged to care about the needs of other people in the local community and around the world

.226 .154 .146 .605 .320 .110

S221 21. I get on well with my teachers .335 .348 .329 .458 -.109 -.005 S223 23. I know that the senior teachers want us to learn successfully .356 .239 .180 .408 .284 -.061

S225 25. My teachers constantly expect me to improve on my personal best

.028 .292 .246 .381 .060 -.008 0.76

S240 40. My parents or carers have always been able to sort out any problems that I have had

.070 .218 .151 .190 .706 .044

S238 38. My parents or carers feel involved with my learning and my progress

.281 .133 .128 .164 .680 .161

S237 37. My parents or carers feel that they are always welcome in the Academy

.144 .135 .292 .309 .569 .169

S239 39. My parents or carers are pleased with my progress .133 .272 .424 .026 .541 .033 0.78 S241 41. I am encouraged to lead activities in the classroom .013 .205 .157 .108 .126 .764

S242 42. I have had opportunities to lead other activities at the Academy

.246 -.085 .032 .141 .044 .668

S231 31. I feel that my successes with activities outside the Academy are recognised

.401 .043 .101 .074 .342 .445 0.64

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization.a. Rotation converged in 10 iterations.

These factors suggest the presence of underlying themes in the data. These could be computed into

new variables which would provide a more economical and interpretable measure, with more

statistical power. Table 26 summarises these themes.

69

Table 34 Themes from the factor analysis

SUMMARY AND FINDINGS FROM STRAND TWO

For Strand Two the questionnaire was designed to assess students' perceptions about their

learning, teaching, progress and achievement in an Oasis Academy.

The findings showed there to be no mean differences in agreement or disagreement for 10 of the

42 items assessing student perceptions on their learning, teaching, progress and achievement in an

Oasis Academy. However, Academy 1 student respondents reported averagely higher mean

responses than their counterparts of Academy 3 to the remaining 32 items which suggests that, to

some degree, students in Academy 1 tend to agree more than Academy 3 students.

With regards to gender differences, the statistical results indicated that male and female student

respondents on average provided similar responses to 20 out of 42 items, although, across the

Academies female student respondents reported significantly higher means than male student

respondents on the remaining 22 items. Thus, there appear to be gender differences on over half

of the items, possibly suggesting a difference in how males and females perceive at least some of

their learning, teaching, progress and achievement.

There were six underlying themes in the data which could be used for a more economical

evaluation and presentation of data. They also suggest that how students perceive their schooling is

consistent with the values of the Oasis Community Learning Charter.

Theme one I have good relationships for learning in my classroom and community

Theme two I enjoy learning and I take responsibility for it

Theme three I am doing as well as I can

Theme four Everybody in my Academy cares about my learning

Theme five My family is involved in my learning

Theme six I have opportunities to exercise leadership inside and outside the Academy

70

STRAND THREE - STORIES OF TRANSFORMATION

Strand Three was a narrative interview with a small sample of students which aimed to explore

whether students were deeply engaged in their learning and whether they were able to recount

stories of significant personal change.

THE SAMPLE

The sample consisted of 22 students in Year 9 from two of the Academies. Thirteen were from

Academy 1 and nine were from Academy 3. Academy 2 data was not available for this analysis.

Table 35 Strand 3 Sample of students by Academy and gender

Gender Total

Male Female

Academy Lords Hill 4 9 13

Wintringham 4 5 9

Total 8 14 22

NARRATIVE INTERVIEW SCHEDULE

Each student was invited to talk with a teacher in response to the following questions.

1. Tell me about a time when you learned something really successfully in the last two or three

weeks

Why did you learn successfully?

What did it feel like

What happened as a result?

2. Tell me about your worst experience of learning in the last two or three weeks?

Why was it a bad experience of learning?

What did it feel like?

What happened as a result?

3. How different are you now as a learner compared with a year ago?

How would you describe the differences?

What has happened to make these differences?

How do you feel now about the future as a learner?

What changes would help you to learn better?

71

4. How would you describe yourself as a learner now? If you could choose an animal (or a car)

which is most like you as a learner, which animal (or car) would it be? Why did you choose

that particular animal (or car)? [Probe further for reasons, if necessary]

5. What have been the best things about your time at the Academy so far?

6. What have been the worst things about your time at the Academy so far?

7. How do you feel about your future, after your time at the Academy?

ANALYSIS

The interviews were recorded and transcribed. The analysis was thematic, based on the criteria for

deep engagement in learning developed in the Learning Futures Projects 2010/11. The key question

was ‘Is deep transformative learning taking place?’ Judgements were based on the identification of

the following themes:

Authenticity –self-authorship and a genuine and meaningful learning process

Identity – a rich language of learning described in terms of learning as a journey or active

process

Agency – taking responsibility for own learning and purposeful application

Stories of significant learning

Each student was rated on a scale of 1 to 4 as follows:

4 = very high evidence of deep transformative learning taking place

3 = high evidence of deep transformative learning taking place

2 = low evidence of deep transformative learning taking place

1 = very low evidence of deep transformative learning taking place

FINDINGS

The following tables show the scores for each student in Academy 1. There was no significant

difference between genders.

Table 36 Rating score for students in Academy 1

Frequency Percent Valid Percent Cumulative Percent

Evidence of deep

transformative

learning taking

place

very low 1 7.7 7.7 7.7

low 5 38.5 38.5 46.2

high 6 46.2 46.2 92.3

very high 1 7.7 7.7 100.0

72

Frequency Percent Valid Percent Cumulative Percent

Evidence of deep

transformative

learning taking

place

very low 1 7.7 7.7 7.7

low 5 38.5 38.5 46.2

high 6 46.2 46.2 92.3

very high 1 7.7 7.7 100.0

Total 13 100.0 100.0

Table 37Histogram of rating score for students in Academy 1

SUMMARY

From this sample we there was only one student who demonstrated very little evidence of

transformative learning in his or her narrative. Overall there was greater evidence of the presence

of transformative learning in the sample with one student ranking very high.

STRAND FOUR - POST-16 TRANSITION AND PROGRESS TO ADULTHOOD

This questionnaire was designed to explore the experiences of leavers from two Academies in

terms of the Oasis Charter, and their expectations of their transition to the next stage of education.

73

SAMPLE

The questionnaire contained 30 items and was administrated in two Academies and in total

received responses from 138 student participants in Year 11.

FINDINGS

The following Table 38 shows the frequency of responses to each question as a number and as a

percentage. The final column shows the missing values.

Table 38 Strand Four frequency of value response per question for Y11 leavers

1

Strongly

Disagree

2

Disagree

3

Agree

4

Strongly

Agree

missing

Item N % N % N % N % N %

S41: I was valued and included during my time at the Academy

0 0.0 7 5.1 99 71.7 32 23.2 0 0.0

S42: I felt part of a community that was focused on learning

2 1.4 9 6.5 101 73.2 26 18.8 0 0.0

S43: I felt safe at the Academy 1 0.7 7 5.1 88 63.8 42 30.4 0 0.0

S44: I enjoyed my time at the Academy 1 0.7 10 7.2 88 63.8 38 27.5 1 0.7

S45: I did as well as I had hoped in my GCSEs

1 0.7 25 18.1 84 60.9 22 15.9 6 4.3

S46: My teachers accurately predicted my GCSE results

3 2.2 27 19.6 80 58.0 22 15.9 6 4.3

S47: I am looking forward to the move to Sixth Form, Further Education, training or work

3 2.2 14 10.1 72 52.2 46 33.3 3 2.2

S48: I have been well prepared for the next stage of my life

1 0.7 14 10.1 90 65.2 29 21.0 4 2.9

S49: My successes were recognised during my time at the Academy

3 2.2 8 5.8 97 70.3 26 18.8 4 2.9

S410: I was helped and supported whenever I struggled with my learning

2 1.4 15 10.9 85 61.6 32 23.2 4 2.9

S411: I developed a sense of responsibility towards other people during my time at the Academy

1 0.7 13 9.4 90 65.2 29 21.0 5 3.6

S412: The Academy helped to make me more understanding about different cultures and types of people

4 2.9 13 9.4 94 68.1 24 17.4 3 2.2

S413: I supported activities that would improve things for other people

2 1.4 20 14.5 96 69.6 17 12.3 3 2.2

S414: I was offered opportunities for leadership during my time at the Academy

5 3.6 23 16.7 84 60.9 23 16.7 3 2.2

S415: I led activities during my time at the Academy

4 2.9 39 28.3 71 51.4 21 15.2 3 2.2

S416: My teachers taught me well at the Academy

2 1.4 10 7.2 77 55.8 45 32.6 4 2.9

S417: My teachers were always trying to improve their teaching

1 0.7 18 13.0 84 60.9 32 23.2 3 2.2

S418: I was regularly challenged to do 1 0.7 16 11.6 89 64.5 28 20.3 4 2.9

74

better so that I could reach my full potential

S419: My parents/carers felt involved with my education at the Academy

7 5.1 23 16.7 80 58.0 22 15.9 6 4.3

S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education

5 3.6 14 10.1 81 58.7 34 24.6 4 2.9

S421: Senior staff at the Academy wanted us to learn well and achieve our best

0 0.0 7 5.1 92 66.7 33 23.9 6 4.3

S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well

1 0.7 15 10.9 87 63.0 26 18.8 9 6.5

S423: The Academy was well run 1 0.7 13 9.4 84 60.9 30 21.7 10 7.2

S424: There were great facilities for learning

0 0.0 9 6.5 83 60.1 36 26.1 10 7.2

S425: There was a broad curriculum that helped me to develop as a whole person

0 0.0 17 12.3 84 60.9 25 18.1 12 8.7

S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum

2 1.4 14 10.1 88 63.8 24 17.4 10 7.2

S427: The Academy made good use of new technologies for learning

1 0.7 9 6.5 87 63.0 32 23.2 9 6.5

S428: I never felt that anyone was putting limits on what I could achieve

2 1.4 15 10.9 84 60.9 28 20.3 9 6.5

S429: I feel proud about what I achieved at the Academy

2 1.4 16 11.6 84 60.9 25 18.1 11 8.0

S430: I could not have gone to a better place for my secondary education

6 4.3 12 8.7 75 54.3 35 25.4 10 7.2

Interestingly, student respondents provided overall positive responses to the majority of the

statements. Four in five (at least 85%) respondents reported agree or strongly agree to 24 out of

30 items (see Table 38). Particularly, 30.4%, 33.6% and 34.1% of the students expressed their strong

agreement on feeling safe when they were in the Academy (S43), their Academy teachers teaching

them well (S416) and looking forward to their future life (S47) respectively. The 6 items that

received the responses of disagree or strongly disagree from 16.3% up to 31.9% of student

respondents (see Table 38) with a mean just below 3.00 (see Table 40) were about lending their

support to activities that ‘would improve things for other people’ (S413), meeting self-expectation

in GCSE (S45), being offered opportunities for leadership (S414), teachers having precisely predicted

what they could achieve in GCSE (S46), parents/carers’ feeling in involvement with ‘my education at

the Academy’ (S419) and having led activates during their time in the Academy (S415).

Table 39 ranks the questions in order of the strength of agreement, combining strongly agree and

agree into one score and strongly disagree and disagree into one score.

Table 39 Rank order of agreement to each question

Item number and description 1+2 Disagree / Strongly Disagree

1 Strongly Disagree

2 Disagree

3 Agree

4 Strongly

Agree

3+4 Agree/

Strongly Agree

Item N % N % N % N % N % N %

S41: I was valued and included during my time at the Academy

7 5.1 0 0.0 7 5.1 99 71.7 32 23.2 131 94.9

75

S421: Senior staff at the Academy wanted us to learn well and achieve our best

7 5.3 0 0.0 7 5.3 92 69.7 33 25.0 125 94.7

S43: I felt safe at the Academy 8 5.8 1 0.7 7 5.1 88 63.8 42 30.4 130 94.2

S424: There were great facilities for learning

9 7.0 0 0.0 9 7.0 83 64.8 36 28.1 119 93.0

S427: The Academy made good use of new technologies for learning

10 7.8 1 0.8 9 7.0 87 67.4 32 24.8 119 92.2

S42: I felt part of a community that was focused on learning

11 8.0 2 1.4 9 6.5 101 73.2 26 18.8 127 92.0

S44: I enjoyed my time at the Academy

11 8.0 1 0.7 10 7.3 88 64.2 38 27.7 126 92.0

S49: My successes were recognised during my time at the Academy

11 8.2 3 2.2 8 6.0 97 72.4 26 19.4 123 91.8

S416: My teachers taught me well at the Academy

12 9.0 2 1.5 10 7.5 77 57.5 45 33.6 122 91.0

S411: I developed a sense of responsibility towards other people during my time at the Academy

14 10.5 1 0.8 13 9.8 90 67.7 29 21.8 119 89.5

S423: The Academy was well run 14 10.9 1 0.8 13 10.2 84 65.6 30 23.4 114 89.1

S48: I have been well prepared for the next stage of my life

15 11.2 1 0.7 14 10.4 90 67.2 29 21.6 119 88.8

S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well

16 12.4 1 0.8 15 11.6 87 67.4 26 20.2 113 87.6

S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum

16 12.5 2 1.6 14 10.9 88 68.8 24 18.8 112 87.5

S47: I am looking forward to the move to Sixth Form, Further Education, training or work

17 12.6 3 2.2 14 10.4 72 53.3 46 34.1 118 87.4

S412: The Academy helped to make me more understanding about different cultures and types of people

17 12.6 4 3.0 13 9.6 94 69.6 24 17.8 118 87.4

S410: I was helped and supported whenever I struggled with my learning

17 12.7 2 1.5 15 11.2 85 63.4 32 23.9 117 87.3

S418: I was regularly challenged to do better so that I could reach my full potential

17 12.7 1 0.7 16 11.9 89 66.4 28 20.9 117 87.3

S428: I never felt that anyone was putting limits on what I could achieve

17 13.2 2 1.6 15 11.6 84 65.1 28 21.7 112 86.8

S425: There was a broad curriculum that helped me to develop as a whole person

17 13.5 0 0.0 17 13.5 84 66.7 25 19.8 109 86.5

S417: My teachers were always trying to improve their teaching

19 14.1 1 0.7 18 13.3 84 62.2 32 23.7 110 85.9

S430: I could not have gone to a better place for my secondary education

18 14.1 6 4.7 12 9.4 75 58.6 35 27.3 116 85.9

S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education

19 14.2 5 3.7 14 10.4 81 60.4 34 25.4 109 85.8

S429: I feel proud about what I achieved at the Academy

18 14.2 2 1.6 16 12.6 84 66.1 25 19.7 115 85.8

S413: I supported activities that would improve things for other

22 16.3 2 1.5 20 14.8 96 71.1 17 12.6 113 83.7

76

people

S45: I did as well as I had hoped in my GCSEs

26 19.7 1 0.8 25 18.9 84 63.6 22 16.7 106 80.3

S414: I was offered opportunities for leadership during my time at the Academy

28 20.7 5 3.7 23 17.0 84 62.2 23 17.0 107 79.3

S46: My teachers accurately predicted my GCSE results

30 22.7 3 2.3 27 20.5 80 60.6 22 16.7 102 77.3

S419: My parents/carers felt involved with my education at the Academy

30 22.7 7 5.3 23 17.4 80 60.6 22 16.7 102 77.3

S415: I led activities during my time at the Academy

43 31.9 4 3.0 39 28.9 71 52.6 21 15.6 92 68.1

The following Table 40 shows the mean score and standard deviation for each question, ranked

from the lowest score. Table 40 Mean score per item, ranked from lowest to highest

Item N Min Max Mean SD

S415: I led activities during my time at the Academy 135 1 4 2.81 0.73

S419: My parents/carers felt involved with my education at the Academy 132 1 4 2.89 0.74

S46: My teachers accurately predicted my GCSE results 132 1 4 2.92 0.68

S414: I was offered opportunities for leadership during my time at the Academy 135 1 4 2.93 0.70

S413: I supported activities that would improve things for other people 135 1 4 2.95 0.58

S45: I did as well as I had hoped in my GCSEs 132 1 4 2.96 0.62

S412: The Academy helped to make me more understanding about different cultures and types of people

135 1 4 3.02 0.63

S429: I feel proud about what I achieved at the Academy 127 1 4 3.04 0.62

S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum

128 1 4 3.05 0.60

S425: There was a broad curriculum that helped me to develop as a whole person 126 2 4 3.06 0.58

S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well

129 1 4 3.07 0.59

S428: I never felt that anyone was putting limits on what I could achieve 129 1 4 3.07 0.63

S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education

134 1 4 3.07 0.71

S418: I was regularly challenged to do better so that I could reach my full potential

134 1 4 3.07 0.60

S430: I could not have gone to a better place for my secondary education 128 1 4 3.09 0.74

S417: My teachers were always trying to improve their teaching 135 1 4 3.09 0.63

S49: My successes were recognised during my time at the Academy 134 1 4 3.09 0.58

S42: I felt part of a community that was focused on learning 138 1 4 3.09 0.55

S48: I have been well prepared for the next stage of my life 134 1 4 3.10 0.59

S410: I was helped and supported whenever I struggled with my learning 134 1 4 3.10 0.64

S411: I developed a sense of responsibility towards other people during my time at the Academy

133 1 4 3.11 0.58

S423: The Academy was well run 128 1 4 3.12 0.60

S427: The Academy made good use of new technologies for learning 129 1 4 3.16 0.57

S41: I was valued and included during my time at the Academy 138 2 4 3.18 0.50

S44: I enjoyed my time at the Academy 137 1 4 3.19 0.59

S47: I am looking forward to the move to Sixth Form, Further Education, training or work

135 1 4 3.19 0.71

S421: Senior staff at the Academy wanted us to learn well and achieve our best 132 2 4 3.20 0.52

S424: There were great facilities for learning 128 2 4 3.21 0.56

S416: My teachers taught me well at the Academy 134 1 4 3.23 0.65

S43: I felt safe at the Academy 138 1 4 3.24 0.57

77

COMPARISONS BETWEEN ACADEMIES

Individual independent t-tests were carried out on each item to examine whether there were

statistically significant mean differences in individual items between the Academies. In general,

there were no statistical mean differences at α = 0.05 level between the two Academies in 24 out of

30 items indicating the student respondents from the Academies had similar views on these

statements. However, Academy 1 had lower means at α = 0.05 level of statistical significance in the

remaining six items than Academy 3 (see purple shaded in the Table 41 below). On average,

Academy 3 student respondents tended to report a greater degree of agreement on enjoying the

time at the Academy (S44), doing well along with GCSE self-expectation (S45), not ever feeling

‘anyone was putting limits on what I could achieve’ (S428), staff wanting them to achieve their best

(S421), their Academy being well managed (S423), and considering their Academy being best place

for their education (S430).

Table 41 Comparison between Academies on each question

t-test for Equality of Means

(purple shaded = significance at α

= 0.05)

t df Sig. (2-tailed)

Mean Diff.

Academy N Mean SD Total N Mean SD

S41 S41: I was valued and included during my time at the Academy

-0.50 136.00 0.62 -0.04 1 52 3.15 0.50 138 3.18 0.50

-0.50 108.40 0.62 -0.04 3 86 3.20 0.50 Total

S42 S42: I felt part of a community that was focused on learning

-0.92 136.00 0.36 -0.09 1 52 3.04 0.48 138 3.09 0.55

-0.97 123.91 0.34 -0.09 3 86 3.13 0.59 Total

S43 S43: I felt safe at the Academy

-1.67 136.00 0.10 -0.17 1 52 3.13 0.53 138 3.24 0.57

-1.73 118.26 0.09 -0.17 3 86 3.30 0.60 Total S44 S44: I enjoyed my time at the Academy

-2.34 135.00 0.02 -0.24 1 51 3.04 0.56 137 3.19 0.59

-2.37 108.50 0.02 -0.24 3 86 3.28 0.59 Total S45 S45: I did as well as I had hoped in my GCSEs

-2.16 130.00 0.03 -0.24 1 46 2.80 0.69 132 2.96 0.62

-2.04 78.82 0.04 -0.24 3 86 3.05 0.57 Total S46 S46: My teachers accurately predicted my GCSE results

-1.37 130.00 0.17 -0.17 1 47 2.81 0.68 132 2.92 0.68

-1.36 94.08 0.18 -0.17 3 85 2.98 0.67 Total

S47 S47: I am looking forward to the move to Sixth Form, Further Education, training or work

-1.21 133.00 0.23 -0.15 1 51 3.10 0.64 135 3.19 0.71

-1.26 117.68 0.21 -0.15 3 84 3.25 0.74 Total

S48 S48: I have been well prepared for the next stage of my life

-0.87 132.00 0.39 -0.09 1 50 3.04 0.57 134 3.10 0.59

-0.88 106.93 0.38 -0.09 3 84 3.13 0.60 Total

S49 S49: My successes were recognised during my time at the Academy

-1.09 132.00 0.28 -0.11 1 51 3.02 0.62 134 3.09 0.58

-1.07 97.95 0.29 -0.11 3 83 3.13 0.56 Total

S410 S410: I was helped and supported whenever I struggled with my learning

-1.37 132.00 0.17 -0.15 1 50 3.00 0.67 134 3.10 0.64

-1.34 95.67 0.18 -0.15 3 84 3.15 0.61 Total

S411 S411: I developed a sense of responsibility towards other people during my time at the

0.19 131.00 0.85 0.02 1 51 3.12 0.55 133 3.11 0.58

0.20 112.79 0.84 0.02 3 82 3.10 0.60 Total

78

Academy

S412 S412: The Academy helped to make me more understanding about different cultures and types of people

-0.88 133.00 0.38 -0.10 1 51 2.96 0.56 135 3.02 0.63

-0.92 118.93 0.36 -0.10 3 84 3.06 0.66 Total

S413 S413: I supported activities that would improve things for other people

-1.03 133.00 0.30 -0.11 1 51 2.88 0.55 135 2.95 0.58

-1.05 111.17 0.30 -0.11 3 84 2.99 0.59 Total

S414 S414: I was offered opportunities for leadership during my time at the Academy

-0.31 133.00 0.76 -0.04 1 51 2.90 0.57 135 2.93 0.70

-0.33 127.02 0.74 -0.04 3 84 2.94 0.77 Total

S415 S415: I led activities during my time at the Academy

-0.77 133.00 0.44 -0.10 1 51 2.75 0.66 135 2.81 0.73

-0.80 118.16 0.42 -0.10 3 84 2.85 0.77 Total

S416 S416: My teachers taught me well at the Academy

-1.32 132.00 0.19 -0.15 1 51 3.14 0.53 134 3.23 0.65

-1.41 126.79 0.16 -0.15 3 83 3.29 0.71 Total

S417 S417: My teachers were always trying to improve their teaching

-1.00 133.00 0.32 -0.11 1 51 3.02 0.65 135 3.09 0.63

-0.99 101.62 0.33 -0.11 3 84 3.13 0.62 Total

S418 S418: I was regularly challenged to do better so that I could reach my full potential

-0.24 132.00 0.81 -0.03 1 51 3.06 0.54 134 3.07 0.60

-0.25 117.42 0.80 -0.03 3 83 3.08 0.63 Total

S419 S419: My parents/carers felt involved with my education at the Academy

0.68 130.00 0.50 0.09 1 51 2.94 0.68 132 2.89 0.74

0.70 117.11 0.49 0.09 3 81 2.85 0.78 Total

S420 S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education

1.05 132.00 0.30 0.13 1 51 3.16 0.61 134 3.07 0.71

1.11 122.92 0.27 0.13 3 83 3.02 0.76 Total

S421 S421: Senior staff at the Academy wanted us to learn well and achieve our best

-3.25 130.00 0.00 -0.29 1 51 3.02 0.47 132 3.20 0.52

-3.32 113.94 0.00 -0.29 3 81 3.31 0.52 Total

S422 S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well

-1.07 127.00 0.29 -0.11 1 50 3.00 0.49 129 3.07 0.59

-1.13 121.88 0.26 -0.11 3 79 3.11 0.64 Total

S423 S423: The Academy was well run

-2.11 126.00 0.04 -0.23 1 50 2.98 0.59 128 3.12 0.60

-2.11 104.65 0.04 -0.23 3 78 3.21 0.59 Total S424 S424: There were great facilities for learning

-1.82 126.00 0.07 -0.18 1 50 3.10 0.46 128 3.21 0.56

-1.93 121.67 0.06 -0.18 3 78 3.28 0.60 Total S425 S425: There was a broad curriculum that helped me to develop as a whole person

-0.67 124.00 0.51 -0.07 1 49 3.02 0.52 126 3.06 0.58

-0.69 113.87 0.49 -0.07 3 77 3.09 0.61 Total

S426 S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum

-0.40 126.00 0.69 -0.04 1 50 3.02 0.47 128 3.05 0.60

-0.44 124.79 0.66 -0.04 3 78 3.06 0.67 Total

S427 S427: The Academy made good use of new technologies for learning

-1.00 127.00 0.32 -0.10 1 50 3.10 0.61 129 3.16 0.57

-0.97 94.46 0.34 -0.10 3 79 3.20 0.54 Total

S428 S428: I never felt that anyone was putting limits on what I could achieve

-2.19 127.00 0.03 -0.24 1 50 2.92 0.63 129 3.07 0.63

-2.17 101.19 0.03 -0.24 3 79 3.16 0.61 Total

S429 S429: I feel proud about what I achieved at the Academy

-1.45 125.00 0.15 -0.16 1 49 2.94 0.63 127 3.04 0.62

-1.44 100.82 0.15 -0.16 3 78 3.10 0.62 Total

S430 S430: I could not have -2.29 126.00 0.02 -0.30 1 49 2.90 0.71 128 3.09 0.74

79

gone to a better place for my secondary education

-2.31 104.68 0.02 -0.30 3 79 3.20 0.74 Total N Mean Std. Deviat

ion

COMPARISON IN ITEM MEAN BETWEEN GENDERS

Individual independent t-test were carried out on each item to examine whether there were

statistically significant mean differences in individual items between genders. Regardless of

Academies, statistical results found no significant gender differences in all statements at α = 0.05

level. In other words, male and female student respondents provided similar responses to all

statements (see Table 42 below).

Table 42 Comparison between genders on each item

t-test for Equality of Means

(purple shaded = significance at α =

0.05)

t df Sig.

(2-

tailed)

Mean

Diff.

GENDER N Mean SD

S41 S41: I was valued and included during my time at the Academy

-0.60 102.00 0.55 -0.06 1 Male 52 3.17 0.51

-0.60 101.19 0.55 -0.06 2 Female 52 3.23 0.47

S42 S42: I felt part of a community that was focused on learning

0.70 102.00 0.49 0.08 1 Male 52 3.17 0.55

0.70 101.88 0.49 0.08 2 Female 52 3.10 0.57

S43 S43: I felt safe at the Academy 0.85 102.00 0.40 0.10 1 Male 52 3.37 0.53

0.85 98.80 0.40 0.10 2 Female 52 3.27 0.63

S44 S44: I enjoyed my time at the Academy 0.78 101.00 0.44 0.09 1 Male 52 3.29 0.50

0.78 90.69 0.44 0.09 2 Female 51 3.20 0.69

S45 S45: I did as well as I had hoped in my GCSEs

0.79 101.00 0.43 0.10 1 Male 52 3.08 0.55

0.79 96.45 0.43 0.10 2 Female 51 2.98 0.68

S46 S46: My teachers accurately predicted my GCSE results

1.64 101.00 0.10 0.21 1 Male 52 3.12 0.62

1.64 98.84 0.10 0.21 2 Female 51 2.90 0.70

S47 S47: I am looking forward to the move to Sixth Form, Further Education, training or work

-0.18 99.00 0.86 -0.03 1 Male 51 3.25 0.74

-0.18 97.36 0.86 -0.03 2 Female 50 3.28 0.64

S48 S48: I have been well prepared for the next stage of my life

1.06 98.00 0.29 0.12 1 Male 50 3.20 0.49

1.06 92.56 0.29 0.12 2 Female 50 3.08 0.63

S49 S49: My successes were recognised during my time at the Academy

2.00 98.00 0.05 0.21 1 Male 51 3.24 0.47

1.99 91.57 0.05 0.21 2 Female 49 3.02 0.59

S410 S410: I was helped and supported whenever I struggled with my learning

0.61 98.00 0.54 0.08 1 Male 50 3.18 0.60

0.61 95.25 0.54 0.08 2 Female 50 3.10 0.71

S411 S411: I developed a sense of responsibility towards other people during my time at the Academy

1.07 97.00 0.29 0.12 1 Male 50 3.22 0.55

1.07 96.93 0.29 0.12 2 Female 49 3.10 0.55

S412 S412: The Academy helped to make me more understanding about different cultures and types of people

-1.09 99.00 0.28 -0.14 1 Male 51 2.96 0.63

-1.09 98.81 0.28 -0.14 2 Female 50 3.10 0.65

S413 S413: I supported activities that would improve things for other people

1.05 99.00 0.29 0.12 1 Male 51 3.08 0.59

1.05 98.22 0.29 0.12 2 Female 50 2.96 0.53

S414 S414: I was offered opportunities for 0.73 99.00 0.47 0.10 1 Male 51 3.02 0.71

80

leadership during my time at the Academy 0.73 98.84 0.47 0.10 2 Female 50 2.92 0.67

S415 S415: I led activities during my time at the Academy

-1.05 99.00 0.30 -0.16 1 Male 51 2.76 0.79

-1.05 97.88 0.30 -0.16 2 Female 50 2.92 0.70

S416 S416: My teachers taught me well at the Academy

1.23 99.00 0.22 0.15 1 Male 51 3.37 0.60

1.23 98.04 0.22 0.15 2 Female 50 3.22 0.65

S417 S417: My teachers were always trying to improve their teaching

0.94 99.00 0.35 0.12 1 Male 51 3.18 0.62

0.94 98.98 0.35 0.12 2 Female 50 3.06 0.62

S418 S418: I was regularly challenged to do better so that I could reach my full potential

0.47 98.00 0.64 0.06 1 Male 50 3.12 0.66

0.47 97.63 0.64 0.06 2 Female 50 3.06 0.62

S419 S419: My parents/carers felt involved with my education at the Academy

0.17 96.00 0.87 0.02 1 Male 50 2.92 0.75

0.17 95.83 0.87 0.02 2 Female 48 2.90 0.69

S420 S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education

-0.27 98.00 0.79 -0.04 1 Male 50 3.08 0.70

-0.27 96.92 0.79 -0.04 2 Female 50 3.12 0.77

S421 S421: Senior staff at the Academy wanted us to learn well and achieve our best

1.75 96.00 0.08 0.18 1 Male 49 3.35 0.52

1.75 95.98 0.08 0.18 2 Female 49 3.16 0.51

S422 S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well

-1.19 96.00 0.24 -0.15 1 Male 50 3.02 0.65

-1.19 94.72 0.24 -0.15 2 Female 48 3.17 0.56

S423 S423: The Academy was well run 0.48 95.00 0.63 0.06 1 Male 49 3.22 0.62

0.48 94.31 0.63 0.06 2 Female 48 3.17 0.56

S424 S424: There were great facilities for learning

1.04 95.00 0.30 0.12 1 Male 49 3.35 0.56

1.04 94.99 0.30 0.12 2 Female 48 3.23 0.56

S425 S425: There was a broad curriculum that helped me to develop as a whole person

-0.59 94.00 0.56 -0.07 1 Male 50 3.06 0.62

-0.59 93.77 0.55 -0.07 2 Female 46 3.13 0.54

S426 S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum

-0.04 95.00 0.97 -0.01 1 Male 50 3.08 0.67

-0.04 94.56 0.97 -0.01 2 Female 47 3.09 0.58

S427 S427: The Academy made good use of new technologies for learning

1.42 96.00 0.16 0.15 1 Male 50 3.32 0.59

1.42 93.45 0.16 0.15 2 Female 48 3.17 0.48

S428 S428: I never felt that anyone was putting limits on what I could achieve

0.28 96.00 0.78 0.04 1 Male 50 3.10 0.74

0.28 91.42 0.78 0.04 2 Female 48 3.06 0.56

S429 S429: I feel proud about what I achieved at the Academy

-0.23 94.00 0.82 -0.03 1 Male 49 3.10 0.55

-0.23 93.97 0.82 -0.03 2 Female 47 3.13 0.54

S430 S430: I could not have gone to a better place for my secondary education

1.75 95.00 0.08 0.26 1 Male 49 3.29 0.79

1.75 94.03 0.08 0.26 2 Female 48 3.02 0.70

COMPARISON IN ITEM MEAN BETWEEN STUDENTS WITH FREE SCHOOL MEALS AND NON-FREE

SCHOOL MEALS

An individual independent t-test was carried out on each item to examine whether there were

statistically significant mean differences in individual items between FSM and non FSM receivers.

However, not all Academy 1 student respondents provided FSM information. Statistical results

found no significant FSM/non FSM differences in all statements at α = 0.05 level. In other words,

FSM and non FSM student respondents in Academy 3 provided similar responses to all statements

(see the Table 43 below).

Table 43 Comparison between students with and without Free School Meals on each item

t df Sig. (2-

tailed)

Mean

Diff.

FSM N Mean SD

S41 S41: I was valued and included during my time at the Academy

0.87 74.00 0.39 0.11 1 FSM 26 3.27 0.60

0.81 41.00 0.43 0.11 2 No FSM 50 3.16 30.47

S42 S42: I felt part of a community that was -1.28 74.00 0.21 -0.18 1 FSM 26 3.04 0.77

81

focused on learning -1.10 34.64 0.28 -0.18 2 No FSM 50 3.22 0.46

S43 S43: I felt safe at the Academy 0.99 74.00 0.33 0.14 1 FSM 26 3.42 0.58

1.01 53.05 0.32 0.14 2 No FSM 50 3.28 0.61

S44 S44: I enjoyed my time at the Academy -1.60 74.00 0.11 -0.23 1 FSM 26 3.15 0.73

-1.42 37.01 0.16 -0.23 2 No FSM 50 3.38 0.49

S45 S45: I did as well as I had hoped in my GCSEs

-0.17 74.00 0.86 -0.02 1 FSM 26 3.08 0.56

-0.17 49.46 0.86 -0.02 2 No FSM 50 3.10 0.54

S46 S46: My teachers accurately predicted my GCSE results

-0.14 74.00 0.89 -0.02 1 FSM 26 3.04 0.77

-0.13 38.51 0.90 -0.02 2 No FSM 50 3.06 0.55

S47 S47: I am looking forward to the move to Sixth Form, Further Education, training or work

0.09 72.00 0.93 0.02 1 FSM 26 3.31 0.84

0.09 40.04 0.93 0.02 2 No FSM 48 3.29 0.62

S48 S48: I have been well prepared for the next stage of my life

-0.39 72.00 0.70 -0.05 1 FSM 26 3.15 0.67

-0.36 40.40 0.72 -0.05 2 No FSM 48 3.21 0.50

S49 S49: My successes were recognised during my time at the Academy

1.16 71.00 0.25 0.16 1 FSM 26 3.27 0.72

1.03 37.27 0.31 0.16 2 No FSM 47 3.11 0.48

S410 S410: I was helped and supported whenever I struggled with my learning

-0.33 72.00 0.74 -0.05 1 FSM 26 3.12 0.77

-0.30 39.74 0.77 -0.05 2 No FSM 48 3.17 0.56

S411 S411: I developed a sense of responsibility towards other people during my time at the Academy

-0.35 70.00 0.73 -0.05 1 FSM 25 3.12 0.73

-0.31 35.51 0.76 -0.05 2 No FSM 47 3.17 0.48

S412 S412: The Academy helped to make me more understanding about different cultures and types of people

-0.38 72.00 0.70 -0.06 1 FSM 26 3.00 0.69

-0.38 49.64 0.71 -0.06 2 No FSM 48 3.06 0.67

S413 S413: I supported activities that would improve things for other people

0.27 72.00 0.79 0.04 1 FSM 26 3.04 0.77

0.23 34.87 0.82 0.04 2 No FSM 48 3.00 0.46

S414 S414: I was offered opportunities for leadership during my time at the Academy

-0.21 72.00 0.83 -0.04 1 FSM 26 2.96 0.87

-0.20 40.51 0.84 -0.04 2 No FSM 48 3.00 0.65

S415 S415: I led activities during my time at the Academy

-0.27 72.00 0.79 -0.05 1 FSM 26 2.85 0.78

-0.26 49.47 0.79 -0.05 2 No FSM 48 2.90 0.75

S416 S416: My teachers taught me well at the Academy

0.45 72.00 0.65 0.08 1 FSM 26 3.35 0.80

0.42 41.16 0.68 0.08 2 No FSM 48 3.27 0.61

S417 S417: My teachers were always trying to improve their teaching

0.97 72.00 0.34 0.15 1 FSM 26 3.23 0.71

0.91 43.15 0.37 0.15 2 No FSM 48 3.08 0.58

S418 S418: I was regularly challenged to do better so that I could reach my full potential

-0.02 71.00 0.98 0.00 1 FSM 25 3.08 0.81

-0.02 36.98 0.99 0.00 2 No FSM 48 3.08 0.58

S419 S419: My parents/carers felt involved with my education at the Academy

0.93 69.00 0.36 0.17 1 FSM 24 3.00 0.88

0.84 35.53 0.41 0.17 2 No FSM 47 2.83 0.64

S420 S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education

1.07 71.00 0.29 0.20 1 FSM 25 3.16 0.90

0.98 38.83 0.33 0.20 2 No FSM 48 2.96 0.68

S421 S421: Senior staff at the Academy wanted us to learn well and achieve our best

1.37 69.00 0.17 0.18 1 FSM 25 3.44 0.65

1.23 36.44 0.23 0.18 2 No FSM 46 3.26 0.44

S422 S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well

1.94 69.00 0.06 0.31 1 FSM 24 3.33 0.64

1.95 46.78 0.06 0.31 2 No FSM 47 3.02 0.64

S423 S423: The Academy was well run 1.46 68.00 0.15 0.22 1 FSM 23 3.39 0.58

1.47 45.04 0.15 0.22 2 No FSM 47 3.17 0.60

S424 S424: There were great facilities for learning

0.19 68.00 0.85 0.03 1 FSM 24 3.33 0.70

0.18 38.23 0.86 0.03 2 No FSM 46 3.30 0.55

S425 S425: There was a broad curriculum that helped me to develop as a whole person

-0.99 67.00 0.33 -0.16 1 FSM 24 3.00 0.72

-0.92 38.18 0.36 -0.16 2 No FSM 45 3.16 0.56

S426 S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum

-0.26 68.00 0.80 -0.05 1 FSM 24 3.04 0.69

-0.26 46.96 0.80 -0.05 2 No FSM 46 3.09 0.69

S427 S427: The Academy made good use of new technologies for learning

0.26 69.00 0.80 0.04 1 FSM 24 3.25 0.68

0.24 36.67 0.81 0.04 2 No FSM 47 3.21 0.51

S428 S428: I never felt that anyone was putting limits on what I could achieve

0.11 69.00 0.91 0.02 1 FSM 24 3.17 0.64

0.11 45.64 0.91 0.02 2 No FSM 47 3.15 0.62

S429 S429: I feel proud about what I achieved 1.57 68.00 0.12 0.22 1 FSM 23 3.30 0.56

82

at the Academy 1.55 42.81 0.13 0.22 2 No FSM 47 3.09 0.54

S430 S430: I could not have gone to a better place for my secondary education

0.62 69.00 0.54 0.12 1 FSM 24 3.29 0.75

0.63 48.55 0.53 0.12 2 No FSM 47 3.17 0.79

PRELIMINARY DATA REDUCTION: EXPLORING UNDERLYING THEMES IN THE WAYS STUDENTS

ANSWER THE QUESTIONS

An exploratory factor analysis was computed to identify any underlying themes in the ways in

which students perceived their time at the Academies. Four themes (see Table 45) were identified

and four items were deleted from the process (Deleted Items: S45, S46, S49, S424).

Table 44 Exploratory Factor Analysis showing item loadings and reliability co-efficients

Component

1 2 3 4 α

S422 S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well

.757 .354 .212 .164

S428 S428: I never felt that anyone was putting limits on what I could achieve

.741 .186 .197 -.228

S410 S410: I was helped and supported whenever I struggled with my learning

.716 .223 .072 .213

S417 S417: My teachers were always trying to improve their teaching

.712 .215 .248 .107

S43 S43: I felt safe at the Academy .665 .199 .099 .294

S423 S423: The Academy was well run .610 .410 .223 .016

S419 S419: My parents/carers felt involved with my education at the Academy

.561 .128 .459 .125

S429 S429: I feel proud about what I achieved at the Academy

.557 .365 .181 .396

S44 S44: I enjoyed my time at the Academy .500 .362 .162 .252

S412 S412: The Academy helped to make me more understanding about different cultures and types of people

.469 .275 .245 .364

S42 S42: I felt part of a community that was focused on learning

.459 .386 .207 .228 0.90

S430 S430: I could not have gone to a better place for my secondary education

.267 .767 .031 .066

S427 S427: The Academy made good use of new technologies for learning

.158 .757 .114 .085

S418 S418: I was regularly challenged to do better so that I could reach my full potential

.215 .645 .239 .174

S411 S411: I developed a sense of responsibility towards other people during my time at the Academy

.191 .643 .364 .286

S420 S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education

.382 .613 -.019 .264

S425 S425: There was a broad curriculum that helped me to develop as a whole person

.262 .604 .472 .134

S41 S41: I was valued and included during my time at the Academy

.411 .592 .105 .211

S421 S421: Senior staff at the Academy wanted us to learn well and achieve our best

.461 .491 .110 .027 0.88

S414 S414: I was offered opportunities for leadership during my time at the Academy

.320 .096 .753 .085

83

S415 S415: I led activities during my time at the Academy .164 .044 .733 .045

S413 S413: I supported activities that would improve things for other people

.016 .213 .704 .382

S426 S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum

.332 .458 .610 -.008 0.76

S48 S48: I have been well prepared for the next stage of my life

.055 .042 .254 .710

S47 S47: I am looking forward to the move to Sixth Form, Further Education, training or work

.150 .223 .028 .673

S416 S416: My teachers taught me well at the Academy .401 .459 .014 .518 0.66

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 7 iterations.

UNDERLYING THEMES IN THE DATA

These four underlying themes (see Table 45) demonstrate how the year eleven leavers perceived

their time at the Academies. It does not necessarily demonstrate the degree to which students

experienced this - for example they could have been not at all proud, or seen their Academy as not

a learning community. These could be computed into new variables for greater economy of data

and statistical power. They are theoretically consistent with the Oasis Charter.

Table 45 Themes from factor analysis

Theme One I was proud to be part of a positive learning community

Theme Two I was continually encouraged to do my best for myself and others

Theme Three I was offered the chance to lead and serve others

Theme Four I am positive about the next stage

SUMMARY

This Strand sought to explore the experiences of leavers from two of the Academies and their

expectations of the transition to the next stage of their education.

When comparing Academies, there were no statistical differences between the two Academies in

24 out of 30 items indicating the student respondents had similar views on these statements.

However, Academy 1 had lower means in the remaining six items than Academy 3, where on

average, Academy 3 respondents tended to report a greater degree of agreement on enjoying their

time at the Academy, suggesting that Academy 3 students overall rating their Academy more highly

than Academy 1 students, although this was just on 6 of the 30 items.

There were no statistically significant gender differences in any of the Academies on how students

reported their experiences of the Academy or their expectations of the transition. There were also

no statistically significant differences between students receiving FSM compared to those with no

FSM, suggesting that neither gender nor socioeconomic background was a particular factor

impacting on students’ experiences of their Academies.

84

STRAND FIVE - QUALITY OF RELATIONSHIPS BETWEEN PARENTS/CARERS AND THE

ACADEMY

This Strand was designed to explore how well parents and carers understand and experience the

ECHO charter’s values in practice.

QUESTIONNAIRE

The parents/carers individual questionnaire was structured in two parts. Part I, was general

questions, in addition to one open-ended question, with 24 items on a four-point Likert-scale

labeled, strongly agree, agree, disagree and strongly disagree, which were coded as 4, 3, 2, and 1

respectively. Similarly, Part II consisted of one open-ended question and another set of six items

using the same Likert-scale and the same coding system as employed in Part I, and was for

parents/carers whose first language was not English to fill in. The total number of parent/carer

respondents was 72 from Academy 1.

FINDINGS

The following Table 46 shows the number of respondents, and the percentage for each value on

each question. It also shows the number and percentage of missing values.

Table 46 Strand Five Individual Item Values

1 Strongly

Disagree

2 Disagree 3 Agree 4 Strongly

Agree

0 missing

Item N % N % N % N % N %

Part I

S51: My child is known and valued as an individual

1 1.4 1 1.4 41 56.9 29 40.3 0 .0

S52: The Academy is providing for my child's particular needs

4 5.6 1 1.4 47 65.3 20 27.8 0 .0

S53: My child feels included in the life of the Academy, in classrooms and in other activities

4 5.6 1 1.4 39 54.2 28 38.9 0 .0

S54: The Academy focuses its work on providing the best opportunities for learning for all students

8 11.1 0 .0 37 51.4 27 37.5 0 .0

S55: My child is taught well 6 8.3 2 2.8 41 56.9 23 31.9 0 .0

S56: My child is being challenged to improve on previous best performance in all areas

5 6.9 1 1.4 40 55.6 24 33.3 2 2.8

S57: My child's progress at the Academy has exceeded our expectations

16 22.2 6 8.3 35 48.6 14 19.4 1 1.4

S58: My child's successes are recognised

9 12.5 3 4.2 33 45.8 26 36.1 1 1.4

85

S59: The Academy ensures my child is well looked after

8 11.1 3 4.2 39 54.2 21 29.2 1 1.4

S510: The Academy helps my child to have a healthy lifestyle

8 11.1 2 2.8 54 75.0 8 11.1 0 .0

S511: There is a good standard of behaviour at the Academy

12 16.7 6 8.3 39 54.2 14 19.4 1 1.4

S512: My child is developing an attitude of helping others

19 26.4 2 2.8 40 55.6 8 11.1 3 4.2

S513: My child is taking opportunities to lead activities

22 30.6 0 .0 36 50.0 12 16.7 2 2.8

S514: My child has kept up a good rate of progress at key transitions

7 9.7 1 1.4 43 59.7 20 27.8 1 1.4

gS515: I get regular and helpful information about my child's progress

14 19.4 3 4.2 31 43.1 24 33.3 0 .0

S516: The Academy responds well to my concerns

5 6.9 6 8.3 37 51.4 22 30.6 2 2.8

S517: I always find staff knowledgeable about my child

15 20.8 2 2.8 29 40.3 26 36.1 0 .0

S518: I always find staff helpful when discussing my child's progress

5 6.9 3 4.2 33 45.8 29 40.3 2 2.8

S519: The Academy helps me to support my child's learning

8 11.1 3 4.2 42 58.3 17 23.6 2 2.8

S520: The Academy is led and managed effectively

5 6.9 1 1.4 42 58.3 21 29.2 3 4.2

S521: My impression is that senior staff focus their work on making the Academy a good place to learn

5 6.9 2 2.8 38 52.8 24 33.3 3 4.2

S522: My impression is that senior staff expect all students to achieve well

4 5.6 1 1.4 38 52.8 28 38.9 1 1.4

S523: My child enjoys their education at the Academy

7 9.7 3 4.2 35 48.6 26 36.1 1 1.4

S524: I would recommend the Academy to another Parent/Carer

7 9.7 3 4.2 30 41.7 30 41.7 2 2.8

Part II

S526: The Academy has been helpful in ensuring that I understand the general information sent to parents/carers

0 .0 0 .0 48 66.7 3 4.2 21 29.2

S527: The Academy has been helpful in ensuring that I understand the information about my child's progress

1 1.4 0 .0 6 8.3 6 8.3 59 81.9

S528: I feel that the Academy understand and takes proper account of the particular issues and sensitivities of my culture

1 1.4 0 .0 4 5.6 5 6.9 62 86.1

S529: If I have a concern I feel confident that I can make contact with relevant staff

1 1.4 0 .0 5 6.9 3 4.2 63 87.5

S530: If I have a concern I feel confident that I can communicate with staff effectively

1 1.4 0 .0 3 4.2 5 6.9 63 87.5

86

S531: I would recommend the Academy to another parent/carer from my community

0 .0 0 .0 4 5.6 4 5.6 64 88.9

PARENT/CARER PARTICIPANTS’ TENDENCY TO INDIVIDUAL ITEMS IN PART I

This questionnaire was conducted in one Academy. No information about gender or whether or not

the parent/carer was a native English speaker was provided. Individual item descriptive means,

calculated based on numeric codes of the responses, were used to examine parent/carer

participants’ tendency to disagree or agree with individual statement items in Part I.

Overall, there were one-third of items with a mean below 3.00 indicating descriptively parent/carer

participants’ general disagreement on these items (see Table 39 below). Among those, 'the five

most disagreed with' statements by the respondents were that parents/carers in general did not

think that their child ‘is developing an attitude of helping others’ (S512), ‘is taking opportunities to

lead activities’ (S513), the progress of their child ‘has exceeded’ their expectations (S57), the

Academy having ‘a good standard of behaviour’ (S511) and the Academy helping them to ‘have a

healthy lifestyle’ (S510). Similar findings were also found in terms of frequency of each of the four-

point Likert scale for individual items. As shown in Table 40, 22.5 to 31.4 percentages of

parent/carer respondents strongly disagreed that their child ‘is developing an attitude of helping

others’ (S512), ‘is taking opportunities to lead activities’ (S513), and the progress of their child ‘has

exceeded’ their expectations (S57). Nevertheless, the findings also indicated that the three

statements were among the five items with least consensus between the parents/carers’ responses

(i.e. highest SD (standard deviation), see Table 47).

In contrast to the findings stated above, the five items which parent/carer respondents reported

highest means (above 3.00) were the agreement on their child ‘is known and valued as an

individual’ (S51), senior staff expecting ‘all students to achieve well’ (S522), their child ‘feels

included’ in the school (S53), finding ‘staff helpful’ in discussing their child’s progress (S518), and

‘would recommend the Academy to another parent/carer’ (S524). The five items also received

relatively greatest percentage in ‘strongly agree’ response (40.3%, 39.4%, 38.9%, 41.4% and 42.9%

respectively, see Table 48). Also the respondents reported most consensuses in agreement on the

first three statements in terms of least SD (see Table 48).

Interestingly, all findings mentioned above seemingly indicate that parents/carers think the

Academy is doing its work well in terms of paying attention to individual students, having high

expectation of all students, and being helpful in discussing about their child’s progress. However,

somehow, the work done by the Academy did not lead to students’ development in helping people,

making progress of expectation (see also Table 48 for at least 91% of the respondents agree or

strongly agree on ‘My child is being challenged to improve on previous best performance in all

areas’ (S56)), good behaviour and healthy lifestyle suggesting an issue of effectiveness.

Nevertheless, the majority of the respondents would recommend the Academy to another

parent/carer.

87

Table 47 below, shows the mean and standard deviation for each question, ranked from the lowest

to the highest.

Table 47 Parent/carer participants’ responses to Part I individual items listed by mean in ascending order

Item N Min Max Mean SD

S512: My child is developing an attitude of helping others 69 1 4 2.54 1.02

S513: My child is taking opportunities to lead activities 70 1 4 2.54 1.11

S57 : My child's progress at the Academy has exceeded our expectations 71 1 4 2.66 1.04

S511 1: There is a good standard of behaviour at the Academy 71 1 4 2.77 0.96

S510: The Academy helps my child to have a healthy lifestyle 72 1 4 2.86 0.76

gS515: I get regular and helpful information about my child's progress 72 1 4 2.90 1.08

S517: I always find staff knowledgeable about my child 72 1 4 2.92 1.11

S519: The Academy helps me to support my child's learning 70 1 4 2.97 0.87

S59: The Academy ensures my child is well looked after 71 1 4 3.03 0.89

S514: My child has kept up a good rate of progress at key transitions 71 1 4 3.07 0.83

S58: My child's successes are recognised 71 1 4 3.07 0.96

S516: The Academy responds well to my concerns 70 1 4 3.09 0.83

S55: My child is taught well 72 1 4 3.13 0.82

S523 : My child enjoys their education at the Academy 71 1 4 3.13 0.89

S520: The Academy is led and managed effectively 69 1 4 3.14 0.77

S52: The Academy is providing for my child's particular needs 72 1 4 3.15 0.71

S54: The Academy focuses its work on providing the best opportunities for learning for all students

72 1 4 3.15 0.90

S521: My impression is that senior staff focus their work on making the Academy a good place to learn

69 1 4 3.17 0.80

S56: My child is being challenged to improve on previous best performance in all areas

70 1 4 3.19 0.79

S524: I would recommend the Academy to another Parent/Carer 70 1 4 3.19 0.92

S518: I always find staff helpful when discussing my child's progress 70 1 4 3.23 0.84

S53: My child feels included in the life of the Academy, in classrooms and in other activities

72 1 4 3.26 0.75

S522: My impression is that senior staff expect all students to achieve well 71 1 4 3.27 0.76

S51: My child is known and valued as an individual 72 1 4 3.36 0.59

The following table presents the frequency of each score as a number and a percentage for each

item in Part I of the questionnaire.

Table 48 Frequency of the parent/carer participants to Part I individual items

1 Strongly

Disagree

2 Disagree 3 Agree 4 Strongly

Agree

N % N % N % N %

S510: The Academy helps my child to have a healthy lifestyle

8 11.1 2 2.8 54 75.0 8 11.1

S512: My child is developing an attitude of helping others

19 27.5 2 2.9 40 58.0 8 11.6

S513: My child is taking opportunities to lead activities 22 31.4 0 .0 36 51.4 12 17.1

88

S511: There is a good standard of behaviour at the Academy

12 16.9 6 8.5 39 54.9 14 19.7

S57: My child's progress at the Academy has exceeded our expectations

16 22.5 6 8.5 35 49.3 14 19.7

S519 : The Academy helps me to support my child's learning

8 11.4 3 4.3 42 60.0 17 24.3

S52: The Academy is providing for my child's particular needs

4 5.6 1 1.4 47 65.3 20 27.8

S514: My child has kept up a good rate of progress at key transitions

7 9.9 1 1.4 43 60.6 20 28.2

S59: The Academy ensures my child is well looked after 8 11.3 3 4.2 39 54.9 21 29.6

S520: The Academy is led and managed effectively 5 7.2 1 1.4 42 60.9 21 30.4

S516 : The Academy responds well to my concerns 5 7.1 6 8.6 37 52.9 22 31.4

S55: My child is taught well 6 8.3 2 2.8 41 56.9 23 31.9

gS515: I get regular and helpful information about my child's progress

14 19.4 3 4.2 31 43.1 24 33.3

S56: My child is being challenged to improve on previous best performance in all areas

5 7.1 1 1.4 40 57.1 24 34.3

S521: My impression is that senior staff focus their work on making the Academy a good place to learn

5 7.2 2 2.9 38 55.1 24 34.8

S517: I always find staff knowledgeable about my child 15 20.8 2 2.8 29 40.3 26 36.1

S523: My child enjoys their education at the Academy 7 9.9 3 4.2 35 49.3 26 36.6

S58: My child's successes are recognised 9 12.7 3 4.2 33 46.5 26 36.6

S54: The Academy focuses its work on providing the best opportunities for learning for all students

8 11.1 0 .0 37 51.4 27 37.5

S53: My child feels included in the life of the Academy, in classrooms and in other activities

4 5.6 1 1.4 39 54.2 28 38.9

S522: My impression is that senior staff expect all students to achieve well

4 5.6 1 1.4 38 53.5 28 39.4

S51: My child is known and valued as an individual 1 1.4 1 1.4 41 56.9 29 40.3

S518 : I always find staff helpful when discussing my child's progress

5 7.1 3 4.3 33 47.1 29 41.4

S524 : I would recommend the Academy to another Parent/Carer

7 10.0 3 4.3 30 42.9 30 42.9

EXPLORING PATTERNS IN PARENT/CARER PARTICIPANTS’ RESPONSES

An exploratory factor analysis was computed in order to explore whether there were any

underlying themes in the data. The factors, items and alpha reliability co-efficient of the resulting

scale is presented in Table 49 below. One item was deleted: S524.

Table 49 Rotated Component Matrix

Component α

Items 1 2 3 4

S522: My impression is that senior staff expect all students to achieve well

.844 .044 .214 .196

S521: My impression is that senior staff focus their work on .751 .220 .379 .086

89

making the Academy a good place to learn

S518: I always find staff helpful when discussing my child's progress

.724 .445 .101 .115

S54: The Academy focuses its work on providing the best opportunities for learning for all students

.706 .307 .186 .316

S56: My child is being challenged to improve on previous best performance in all areas

.693 .059 .393 .201

S55: My child is taught well .638 .490 .169 .139

S517: I always find staff knowledgeable about my child .635 .466 .112 .349

S510: The Academy helps my child to have a healthy lifestyle .587 .234 .228 .408

gS515: I get regular and helpful information about my child's progress

.466 .349 .320 .444

S519: The Academy helps me to support my child's learning .437 .375 .355 .209 0.93

S51: My child is known and valued as an individual .211 .773 .308 .133

S520: The Academy is led and managed effectively .510 .617 .253 .061

S512: My child is developing an attitude of helping others .198 .608 .117 .270

S53: My child feels included in the life of the Academy, in classrooms and in other activities

.225 .571 .510 .264

S523: My child enjoys their education at the Academy .464 .470 .398 .362 0.85

S511: There is a good standard of behaviour at the Academy .285 .108 .702 .094

S59 : The Academy ensures my child is well looked after .201 .234 .658 .430

S52: The Academy is providing for my child's particular needs .197 .511 .655 .231

S58: My child's successes are recognised .540 .425 .541 -.099 0.82

S514: My child has kept up a good rate of progress at key transitions

.247 .131 .314 .772

S513: My child is taking opportunities to lead activities -.064 .482 -.054 .710

S57: My child's progress at the Academy has exceeded our expectations

.430 -.025 .238 .657

S516: The Academy responds well to my concerns .483 .361 .085 .510 0.82

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization.Rotation converged in 13 iterations.

UNDERLYING THEMES IN THE DATA

There were four themes that emerged from the parents and carers data (see Table 50). These

suggest that the ways in which parents understood their relationship with the Academy were

similar conceptually. The themes could be computed into new variables for greater statistical

power and economy of data.

Table 50 Themes from factor analysis for Strand 5

Theme One The Academy provides an environment that challenges my child and

enables me to help

Theme Two My child enjoys learning as a valued individual in a well run, inclusive

90

Academy

Theme Three The Academy ensures that my child is valued as an individual

Theme Four My child is making good progress in learning and leadership in a responsive

Academy

SUMMARY AND FINDINGS FROM STRAND FIVE

This Strand explored how well parents and carers understood and experienced the ECHO charter’s

values in practice.

The findings showed that a third of the statements within the 30 item 2-part questionnaire were

generally disagreed with. Many of these statements seemed to refer to issues of developing

attitudes to help others or to take opportunities to lead activities or referred to behaviour or a

healthy lifestyle. However, these findings also showed these same statements to be among those

in which the parents/carers had least consensus. This suggests that whilst there are general

disagreements about certain aspects of how parents/carers experienced the Oasis Charter, these

disagreements do not all concord with one another and could reflect the diversity of experiences

due to the parents’ expectations or perceptions. These could be affected by other factors such as

culture or language, however data on English as an additional language was missing so such

assumptions could be problematic.

The underlying themes in the data suggest key ways in which parents approach the Academy, in a

generally positive, inclusive way, where they feel their child is valued and challenged in learning.

STRAND SIX - LEARNING OF TEACHERS: IMPACT OF CPD ON THE QUALITY OF

CLASSROOM PRACTICE

Strand Six was designed to explore teacher’s perceptions of their Academy and the impact of their

professional learning on classroom practice.

QUESTIONNAIRE

The questionnaire was administered in one Academy and received responses from 10 teacher

participants. The questionnaire was in three parts, with questions answered on a four-point Likert-

scale: 28 (Part A), 26 (Part B), 12 (Part C). The following Tables 51, 52 and 53 show the distribution

of scores for each question, as a number and a percentage, as well as missing values.

Table 51 Strand Five Part A Item Values

91

Part A 1 2 3 4 Missing

Item N % N % N % N % N %

S6A1 S6 A1: Staff as well as students learn in this Academy

2 20.0 1 10.0 4 40.0 3 30.0 0 0.0

S6A2 S6 A2: I draw on good practice from other schools as a means to further my own professional development

4 40.0 4 40.0 2 20.0 0 0.0 0 0.0

S6A3 S6 A3: I read research reports as one source of useful ideas for improving my practice

1 10.0 7 70.0 2 20.0 0 0.0 0 0.0

S6A4 S6 A4: I use the web as one source of useful ideas for improving my practice

0 0.0 3 30.0 5 50.0 2 20.0 0 0.0

S6A5 S6 A5: I consult my students about how they learn most effectively

0 0.0 5 50.0 2 20.0 3 30.0 0 0.0

S6A6 S6 A6: I relate what works in my own practice to research findings

0 0.0 6 60.0 3 30.0 1 10.0 0 0.0

S6A7 S6 A7: I am able to see how practices that work in one context might be adapted to other contexts

0 0.0 2 20.0 5 50.0 3 30.0 0 0.0

S6A8 S6 A8: I use insights from my own professional learning to feed into Academy policy development

3 30.0 3 30.0 0 0.0 3 30.0 1 10.0

S6A9 S6 A9: I reflect on my practice as a way of identifying my professional learning needs

0 0.0 0 0.0 7 70.0 3 30.0 0 0.0

S6A10 S6 A10: I experiment with my practice as a conscious strategy for improving classroom teaching and learning

0 0.0 0 0.0 4 40.0 6 60.0 0 0.0

S6A11 S6 A11: I modify my practice in the light of feedback from my students

0 0.0 0 0.0 7 70.0 3 30.0 0 0.0

S6A12 S6 A12: I modify my practice in the light of published research evidence

3 30.0 3 30.0 4 40.0 0 0.0 0 0.0

S6A13 S6 A13: I modify my practice in the light of evidence from self-evaluations of my classroom practice

0 0.0 0 0.0 7 70.0 3 30.0 0 0.0

S6A14 S6 A14: I modify my practice in the light of evidence from evaluations of my classroom practice by managers or other colleagues

0 0.0 0 0.0 4 40.0 5 50.0 1 10.0

S6A15 S6 A15: I carry out joint research/evaluation with one or more colleagues as a way of improving my practice

5 50.0 1 10.0 3 30.0 0 0.0 1 10.0

S6A16 S6 A16: I collaborate to plan my teaching

0 0.0 4 40.0 3 30.0 3 30.0 0 0.0

S6A17 S6 A17: I observe other colleagues in the classroom and we give each other feedback

0 0.0 5 50.0 3 30.0 2 20.0 0 0.0

S6A18 S6 A18: I engage in team teaching as a way of improving practice

3 30.0 4 40.0 1 10.0 2 20.0 0 0.0

S6A19 S6 A19: If I have a problem with my teaching I turn to colleagues for help

0 0.0 0 0.0 8 80.0 2 20.0 0 0.0

S6A20 S6 A20: I suggest ideas or approaches for colleagues to try in class

2 20.0 0 0.0 6 60.0 2 20.0 0 0.0

S6A21 S6 A21: I make collective agreements to test out new ideas

0 0.0 5 50.0 2 20.0 3 30.0 0 0.0

S6A22 S6 A22: I discuss openly with colleagues what and how they are learning

0 0.0 1 10.0 4 40.0 4 40.0 1 10.0

S6A23 S6 A23: I use informal opportunities to discuss how children learn

0 0.0 5 50.0 2 20.0 3 30.0 0 0.0

92

S6A24 S6 A24: I offer reassurance and support to colleagues

0 0.0 2 20.0 2 20.0 6 60.0 0 0.0

S6A25 S6 A25: My practice is based on the belief that all students are capable of learning

0 0.0 0 0.0 2 20.0 8 80.0 0 0.0

S6A26 S6 A26: My experience is that students in this Academy enjoy learning

0 0.0 3 30.0 5 50.0 1 10.0 1 10.0

S6A27 S6 A27: My experience is that student success is celebrated

0 0.0 0 0.0 5 50.0 2 20.0 3 30.0

S6A28 S6 A28: I discuss with colleagues how students might be helped to learn how to learn

0 0.0 0 0.0 6 60.0 3 30.0 1 10.0

Table 52 Strand 6 Part B Item values

Part B 1 2 3 4 Missing

Item N % N % N % N % N %

S6B1 S6 B1: Senior leaders communicate a clear vision of where the Academy is going

3 30.0 2 20.0 2 20.0 3 30.0 0 0.0

S6B2 S6 B2: Staff have a commitment to the whole Academy as well as to their department, key stage and/or year group

0 0.0 0 0.0 4 40.0 6 60.0 0 0.0

S6B3 S6 B3: Senior leaders promote commitment among staff to the whole Academy as well as to the department, key stage and/or year group

0 0.0 2 20.0 5 50.0 3 30.0 0 0.0

S6B4 S6 B4: There is effective communication between senior leaders and teachers

0 0.0 8 80.0 1 10.0 0 0.0 1 10.0

S6B5 S6 B5: There are processes for involving all staff in decision-making

2 20.0 5 50.0 1 10.0 1 10.0 1 10.0

S6B6 S6 B6: Teachers’ professional know-how is used in the formulation of Academy policy and goals

0 0.0 7 70.0 2 20.0 0 0.0 1 10.0

S6B7 S6 B7: Teachers’ professional know-how is used in the formulation of Academy policy, even where this leads to a questioning of established rules, procedures and practices

0 0.0 5 50.0 3 30.0 2 20.0 0 0.0

S6B8 S6 B8: Opportunities are provided for teachers to critically evaluate Academy policy

2 20.0 5 50.0 1 10.0 1 10.0 1 10.0

S6B9 S6 B9: Staff are involved in evaluating Academy policy

3 30.0 4 40.0 0 0.0 3 30.0 0 0.0

S6B10 S6 B10: Staff participate in important decision-making

4 40.0 3 30.0 1 10.0 1 10.0 1 10.0

S6B11 S6 B11: There are processes for involving students in decision-making

0 0.0 8 80.0 1 10.0 0 0.0 1 10.0

S6B12 S6 B12: Staff have a good working knowledge of the Academy Development Plan (or your Academy’s equivalent document)

3 30.0 2 20.0 4 40.0 1 10.0 0 0.0

S6B13 S6 B13: Staff see the Academy Development Plan as relevant and useful to learning and teaching

0 0.0 3 30.0 5 50.0 2 20.0 0 0.0

S6B14 S6 B14: Staff development time is used effectively to realise Academy Development Plan priorities

0 0.0 5 50.0 4 40.0 1 10.0 0 0.0

S6B15 S6 B15: Staff development time is used effectively in the Academy

0 0.0 5 50.0 5 50.0 0 0.0 0 0.0

93

S6B16 S6 B16: The Academy provides cover to allow staff joint planning time

9 90.0 1 10.0 0 0.0 0 0.0 0 0.0

S6B17 S6 B17: Teachers are encouraged to experiment with new ideas as a way of promoting professional growth

0 0.0 0 0.0 5 50.0 5 50.0 0 0.0

S6B18 S6 B18: Formal training provides opportunities for teachers to develop professionally

0 0.0 2 20.0 7 70.0 1 10.0 0 0.0

S6B19 S6 B19: Teachers are helped to develop skills to assess students’ work in ways that move students on in their learning

0 0.0 3 30.0 5 50.0 2 20.0 0 0.0

S6B20 S6 B20: Teachers are helped to develop skills to observe learning as it happens in the classroom

0 0.0 3 30.0 6 60.0 0 0.0 1 10.0

S6B21 S6 B21: Senior leaders support teachers in sharing practice with other schools through networking

3 30.0 2 20.0 5 50.0 0 0.0 0 0.0

S6B22 S6 B22: Information is collected from teachers on those aspects of their work that they themselves think they do effectively

2 20.0 5 50.0 3 30.0 0 0.0 0 0.0

S6B23 S6 B23: Information is collected from teachers on effective ways they promote learning to learn skills and knowledge among their students

2 20.0 0 0.0 6 60.0 2 20.0 0 0.0

S6B24 S6 B24: Information is collected from teachers on informal teacher networking in which they play an active role

5 50.0 5 50.0 0 0.0 0 0.0 0 0.0

S6B25 S6 B25: Teacher-initiated networking is an integral element of staff development

5 50.0 5 50.0 0 0.0 0 0.0 0 0.0

S6B26 S6 B26: Learning how to learn is an issue discussed in staff development time

2 20.0 4 40.0 3 30.0 0 0.0 1 10.0

Table 53 Strand 6 Part C Item Values

Part C 1 2 3 4 missing

Item N % N % N % N % N %

S6C1 S6 C1: I feel included in a community that values what I can offer to the staff team and to students

0 0.0 2 20.0 4 40.0 4 40.0 0 0.0

S6C2 S6 C2: I believe that I am working in a community that focuses our attention and activities on learning to improve teaching and student outcomes

0 0.0 3 30.0 5 50.0 2 20.0 0 0.0

S6C3 S6 C3: My experience suggests that other members of staff are involved in training and development activities to improve their classroom practice and student outcomes

0 0.0 5 50.0 4 40.0 1 10.0 0 0.0

S6C4 S6 C4: I have had opportunities to work alongside and/or train with colleagues in other phases to improve students’ progress

3 30.0 0 0.0 0 0.0 7 70.0 0 0.0

S6C5 S6 C5: My experience is that senior leaders are role models as ‘leaders of learning’ in the Academy

0 0.0 5 50.0 0 0.0 5 50.0 0 0.0

S6C6 S6 C6: Leadership is shared in this Academy and opportunities to take on leadership roles are available to all those that want them and have the relevant knowledge and experience

0 0.0 1 10.0 8 80.0 0 0.0 1 10.0

94

S6C7 S6 C7: My experience is that the Academy promotes a balance between the rights and the responsibilities of all members of the community

0 0.0 6 60.0 4 40.0 0 0.0 0 0.0

S6C8 S6 C8: There is an emphasis among students and staff on caring about and helping other people in the Academy community

0 0.0 6 60.0 1 10.0 3 30.0 0 0.0

S6C9 S6 C9: There is an emphasis among students and staff on caring about and helping other people in the wider community

0 0.0 5 50.0 2 20.0 3 30.0 0 0.0

S6C10 S6 C10: The Academy is successful in helping parents/carers to support the learning of their children

0 0.0 7 70.0 3 30.0 0 0.0 0 0.0

S6C11 S6 C11: The achievements of staff are recognised and celebrated

0 0.0 4 40.0 5 50.0 0 0.0 1 10.0

S6C12 S6 C12: I believe that I am realising my full potential as a teacher

0 0.0 0 0.0 5 50.0 5 50.0 0 0.0

PART A – ITEM MEAN AND STANDARD DEVIATIONS

There were 16 out of 28 items with a mean of below 3.00 suggesting that the teacher respondents

in the Academy had a tendency to disagree with these statements concerning their professional

learning.

On average, the five least frequently reported processes were concerned with active use of

educational evidence to advance practice and enhance their professional development in term of

‘carry out joint research/evaluation’ to improve practice (S6A15), ‘draw on good practice from

other schools’ for further professional development (S6A2), ‘read research reports’ for practice

improvement (S6A3), ‘modify practice in the light of published research evidence’ (S6A12) and

‘engage in team teaching’ for improvement of practice (S6A18).

In contrast, among the 12 out of 28 items with a mean of above 3.00 the five most regular

processes engaged in by the teacher participants were in relation to improving their teaching and

learning practice based on ‘the belief that all students are capable of learning’ (S6A25), ‘experiment

with my practice as a conscious strategy’ (S6A10), ‘evidence from evaluations of my classroom

practice by managers or other colleagues’ (S6A14), offering ‘reassurance and support to colleagues’

(S6A24), and discussing with colleagues ‘what and how they are learning’ (S6A22) as well as ‘how

students might be helped to learn how to learn’ (S6A28).

Further, there was generally high variation between the teacher respondents’ responses to the

frequency with which they engaged in the following processes: using insights of their professional

learning to ‘feed into Academy policy development’ (S6A8), believing that ‘staff as well as students

learn in this Academy’ (S6A1), being involved in ‘team teaching as a way of improving practice’

(S6A18), suggesting ‘ideas or approaches for colleagues to try in class’ (S6A20) and conducting joint

research/evaluation to improve practice (S6A15). On the other hand, there was a common

agreement by the teacher respondents about themselves, believing ‘all students are capable of

learning’ (S6A25), seeking colleagues’ help when facing ‘a problem with my teaching’ (S6A19),

95

reflecting on their practice for ‘identifying my professional learning needs’ (S6A9), modifying their

practice taking account of ‘evidence from self-evaluation’ (S6A13) as well as ‘feedback from my

students’ (S6A11).

The following Table 54 shows the mean score and standard deviation on each question ranked from

the lowest score to the highest.

Table 54 Strand Six Part A, Means and Standard Deviations, ranked from lowest to highest

N Min Max Mean SD

Part A

S6A15 S6 A15: I carry out joint research/evaluation with one or more colleagues as a way of improving my practice

9 1 3 1.78 0.97

S6A2 S6 A2: I draw on good practice from other schools as a means to further my own professional development

10 1 3 1.80 0.79

S6A3 S6 A3: I read research reports as one source of useful ideas for improving my practice

10 1 3 2.10 0.57

S6A12 S6 A12: I modify my practice in the light of published research evidence

10 1 3 2.10 0.88

S6A18 S6 A18: I engage in team teaching as a way of improving practice 10 1 4 2.20 1.14

S6A8 S6 A8: I use insights from my own professional learning to feed into Academy policy development

9 1 4 2.33 1.32

S6A6 S6 A6: I relate what works in my own practice to research findings 10 2 4 2.50 0.71

S6A17 S6 A17: I observe other colleagues in the classroom and we give each other feedback

10 2 4 2.70 0.82

S6A26 S6 A26: My experience is that students in this Academy enjoy learning

9 2 4 2.78 0.67

S6A5 S6 A5: I consult my students about how they learn most effectively 10 2 4 2.80 0.92

S6A21 S6 A21: I make collective agreements to test out new ideas 10 2 4 2.80 0.92

S6A23 S6 A23: I use informal opportunities to discuss how children learn 10 2 4 2.80 0.92

S6A20 S6 A20: I suggest ideas or approaches for colleagues to try in class 10 1 4 2.80 1.03

S6A1 S6 A1: Staff as well as students learn in this Academy 10 1 4 2.80 1.14

S6A4 S6 A4: I use the web as one source of useful ideas for improving my practice

10 2 4 2.90 0.74

S6A16 S6 A16: I collaborate to plan my teaching 10 2 4 2.90 0.88

S6A7 S6 A7: I am able to see how practices that work in one context might be adapted to other contexts

10 2 4 3.10 0.74

S6A19 S6 A19: If I have a problem with my teaching I turn to colleagues for help

10 3 4 3.20 0.42

S6A27 S6 A27: My experience is that student success is celebrated 7 3 4 3.29 0.49

S6A9 S6 A9: I reflect on my practice as a way of identifying my professional learning needs

10 3 4 3.30 0.48

S6A13 S6 A13: I modify my practice in the light of evidence from self-evaluations of my classroom practice

10 3 4 3.30 0.48

S6A11 S6 A11: I modify my practice in the light of feedback from my students

10 3 4 3.30 0.48

S6A22 S6 A22: I discuss openly with colleagues what and how they are learning

9 2 4 3.33 0.71

S6A28 S6 A28: I discuss with colleagues how students might be helped to learn how to learn

9 3 4 3.33 0.50

96

S6A24 S6 A24: I offer reassurance and support to colleagues 10 2 4 3.40 0.84

S6A14 S6 A14: I modify my practice in the light of evidence from evaluations of my classroom practice by managers or other colleagues

9 3 4 3.56 0.53

S6A10 S6 A10: I experiment with my practice as a conscious strategy for improving classroom teaching and learning

10 3 4 3.60 0.52

S6A25 S6 A25: My practice is based on the belief that all students are capable of learning

10 3 4 3.80 0.42

PART B – ITEM MEAN SCORES AND STANDARD DEVIATIONS

Only three (11.5%) out of 26 items have a mean of above 3.00 indicating that the teacher

respondents considered that the majority of the activities (88.5%) stated in Part B regarding

leadership practices and systems in the Academy were not frequently happening. The three

generally most common processes were about staff having ‘a commitment to the whole Academy

as well as to their department, key stage and/or year group’ (S6B2) along with senior leaders’

promotion of this perspective (S6B3) and teachers being encouraged to ‘experiment with new ideas

as a way of promoting professional growth’ (S6B17). On the contrary, there were four items with a

mean of below 2.00 implying that the most infrequent processes were perceived by these teachers

to be the Academy providing ‘cover to allow staff joint planning time’ (S6B16), teachers collecting

information from informal networking where ‘they play an active role’ (S6B24), teacher-initiated

networking playing an vital role for staff development (S6B25), staff being involved in ‘important

decision-making’ (S6B10) and teachers searching for information about how to do work effectively

(S6B22).

Additionally, the teacher respondents had five greatest disagreement, or variation in response

about how often senior leaders communicate clearly about the vision of the Academy (S6B1), staff

being engaged in evaluating the Academy policy (S6B9), staff having enough understanding of the

Academy plan (S6B12), staff taking part in important decision-making (S6B10) and teachers

collecting information about the effective learning to learn of their students (S6B23), although

these activities were generally reported to tend to be less frequent. Comparatively, the five

processes that received greatest agreement on the responses from the teacher respondents

included how often the Academy buys out staff time for joint planning (S6B16), an effective

communication between senior leaders and teachers taking place (S6B4), students being involved

in the Academy decision-making (S6B11), the Academy using teachers’ profession know-how to

establish relevant policy and goals (S6B6) and helping teachers to develop skills of observing

classroom learning (S6B20).

Table 55 below shows the means and standard deviations for each item in Part B, ranked from

lowest to highest.

97

Table 55 Strand 6, part B, means and standard deviations ranked from lowest to highest

N Min Max Mean SD

Part B

S6B16: The Academy provides cover to allow staff joint planning time 10 1 2 1.10 0.32

S6B24: Information is collected from teachers on informal teacher networking in which they play an active role

10 1 2 1.50 0.53

S6B25: Teacher-initiated networking is an integral element of staff development

10 1 2 1.50 0.53

S6B10: Staff participate in important decision-making 9 1 4 1.89 1.05

S6B22: Information is collected from teachers on those aspects of their work that they themselves think they do effectively

10 1 3 2.10 0.74

S6B4: There is effective communication between senior leaders and teachers

9 2 3 2.11 0.33

S6B26: Learning how to learn is an issue discussed in staff development time

9 1 3 2.11 0.78

S6B8: Opportunities are provided for teachers to critically evaluate Academy policy

9 1 4 2.11 0.93

S6B5: There are processes for involving all staff in decision-making 9 1 4 2.11 0.93

S6B11: There are processes for involving students in decision-making 9 2 3 2.11 0.33

S6B21: Senior leaders support teachers in sharing practice with other schools through networking

10 1 3 2.20 0.92

S6B6: Teachers’ professional know-how is used in the formulation of Academy policy and goals

9 2 3 2.22 0.44

S6B9 : Staff are involved in evaluating Academy policy 10 1 4 2.30 1.25

S6B12: Staff have a good working knowledge of the Academy Development Plan (or your Academy’s equivalent document)

10 1 4 2.30 1.06

S6B15: Staff development time is used effectively in the Academy 10 2 3 2.50 0.53

S6B1 : Senior leaders communicate a clear vision of where the Academy is going

10 1 4 2.50 1.27

S6B14: Staff development time is used effectively to realise Academy Development Plan priorities

10 2 4 2.60 0.70

S6B20: Teachers are helped to develop skills to observe learning as it happens in the classroom

9 2 3 2.67 0.50

S6B7: Teachers’ professional know-how is used in the formulation of Academy policy, even where this leads to a questioning of established rules, procedures and practices

10 2 4 2.70 0.82

S6B23: Information is collected from teachers on effective ways they promote learning to learn skills and knowledge among their students

10 1 4 2.80 1.03

S6B13: Staff see the Academy Development Plan as relevant and useful to learning and teaching

10 2 4 2.90 0.74

S6B19: Teachers are helped to develop skills to assess students’ work in ways that move students on in their learning

10 2 4 2.90 0.74

S6B18: Formal training provides opportunities for teachers to develop professionally

10 2 4 2.90 0.57

S6B3: Senior leaders promote commitment among staff to the whole Academy as well as to the department, key stage and/or year group

10 2 4 3.10 0.74

S6B17: Teachers are encouraged to experiment with new ideas as a way of promoting professional growth

10 3 4 3.50 0.53

S6B2: Staff have a commitment to the whole Academy as well as to their 10 3 4 3.60 0.52

98

department, key stage and/or year group

PART C – ITEM MEANS AND STANDARD DEVIATIONS

There were four out of 12 items with a mean of equal to or above 3.00 indicating that teachers

agree strongly with the belief that they are realizing their ‘full potential as a teacher’ (S6C12),

feeling ‘included in a community that values’ what they can offer (S6C1), being given ‘opportunities

to work alongside and/or train with colleagues in other phases to improve students’ progress’

(S6C4) and experiencing that senior teachers behave as role models ‘as leaders of learning’ (S6C5).

However, the last two questions also demonstrated greatest variation from the teacher

respondents. In contrast, the five items most strongly disagreed with by the teacher respondents

were about the Academy productively assisting ‘parents/carers to support the learning of their

children’ (S6C10), the Academy promoting ‘a balance between the rights and the responsibilities’

within the community (S6C7), staff achievement being ‘recognised and celebrated’ (S6C11), other

staff members ‘involved in training and development activities to improve their classroom practice

and student outcomes’ (S6C3) and ‘students and staff on caring about and helping other people in

the Academy community’ being emphasised (S6C8) where the first three also obtained highest

agreements in the teacher respondents’ responses.

The following Table 56 shows the mean scores and standard deviations, ranked from lowest to

highest.

Table 56 Strand 6 Part C, mean score and standard deviation ranked from lowest to highest

N Min Max Mean SD

Part C

S6C10 S6 C10: The Academy is successful in helping parents/carers to support the learning of their children

10 2 3 2.30 0.48

S6C7 S6 C7: My experience is that the Academy promotes a balance between the rights and the responsibilities of all members of the community

10 2 3 2.40 0.52

S6C11 S6 C11: The achievements of staff are recognised and celebrated 9 2 3 2.56 0.53

S6C3 S6 C3: My experience suggests that other members of staff are involved in training and development activities to improve their classroom practice and student outcomes

10 2 4 2.60 0.70

S6C8 S6 C8: There is an emphasis among students and staff on caring about and helping other people in the Academy community

10 2 4 2.70 0.95

S6C9 S6 C9: There is an emphasis among students and staff on caring about and helping other people in the wider community

10 2 4 2.80 0.92

S6C6 S6 C6: Leadership is shared in this Academy and opportunities to take on leadership roles are available to all those that want them and have the relevant knowledge and experience

9 2 3 2.89 0.33

S6C2 S6 C2: I believe that I am working in a community that focuses our attention and activities on learning to improve teaching and student outcomes

10 2 4 2.90 0.74

99

S6C5 S6 C5: My experience is that senior leaders are role models as ‘leaders of learning’ in the Academy

10 2 4 3.00 1.05

S6C4 S6 C4: I have had opportunities to work alongside and/or train with colleagues in other phases to improve students’ progress

10 1 4 3.10 1.45

S6C1 S6 C1: I feel included in a community that values what I can offer to the staff team and to students

10 2 4 3.20 0.79

S6C12 S6 C12: I believe that I am realising my full potential as a teacher 10 3 4 3.50 0.53

SUMMARY AND FINDINGS FROM STRAND SIX

This Strand was a three part questionnaire designed to explore teachers’ perceptions of their

Academy and the impact of their professional learning on their classroom practice.

The first part of the questionnaire revealed a range of findings with regards to professional learning

processes, particularly research and evaluation and or applying findings of published research to

the teachers’ practice. However, teachers agreed on the five most regular processes engaged in by

the teacher participants which referred to improving their teaching and learning practice.

There was, however, a higher degree of variation with regard to the extent to which they felt

involved or engaged in practices which promoted the policy or professional development of the

Academy. But interestingly, teachers tended to show agreement on items which referred to self-

reflection and evaluation or for seeking help for teaching related issues.

The part of the questionnaire which contained items on leadership practices found that the

majority of statements did not frequently apply to practices at the Academy. Much of the findings

from this section seem to suggest that teachers did not feel especially involved in practices which

might promote the staff as part of a whole team, however, staff reported strong commitment to

their departments, key stage or year group.

The findings, nevertheless, did show that teachers felt they were reaching their potential as

teachers and felt presented with opportunities for development with good role models of senior

teachers. What is also interesting is that teachers generally disagreed with the extent to which the

Academy supports parents in the learning of their children and the balancing of

rights/responsibilities within the community. This seems to concur with the findings of Strand Five

where parents/carers responded in a similar way on items which explored how the Academy

performed as part of a community.

STRAND SEVEN - IMPACT OF CONTINUED PROFESSIONAL DEVELOPMENT (CPD) ON THE

QUALITY OF LEADERSHIP AND MANAGEMENT AND CLASSROOM PRACTICE

This Strand explored senior leaders’ perceptions of how the Academy is fulfilling the OCL Charter.

100

QUESTIONNAIRE

The Leadership questionnaire consisted of 55 items including one open-ended question. The 54

closed items were structured with a four-point Likert type scale labeled, strongly disagree, disagree,

agree, and strongly agree, which were subsequently coded as 1, 2, 3 and 4 respectively and treated

as numerical scores at the stage of analysis in order to investigate respondents’ disagreement or

agreement on individual statements.

SAMPLE AND DATA COLLECTION

The questionnaire was administered in the three Academies as an online survey constructed on the

Academies intranet. In total, 25 questionnaires filled in by SMT staff were received: nine from

Academy 1, ten from Academy 2 and six from Academy 3. The percentages of missing responses

from the 25 respondents ranged from 0.0 to 36.0 across all Likert-scale items. Referring to the

highest missing response rate, there were 9 out of 25 SMT staff (36.0% of the respondents) missing

out reporting whether or not they agree on their lessons being rated as good or outstanding (S73).

Noticeably, of these, 7 were from the same Academy indicating possibly that in the Academy either

SMT staff did not undertake teaching or there was no mechanism of appraisal of SMT staff’s

teaching. Overall SMT participants responded more positively than negatively to two-third of the

items. (Details of SMT participants’ responses to individual items are provided in Appendix 4)

FINDINGS AND DISCUSSION

Table 57 below presents the responses for each question as a numerical value and a percentage. It

also identifies the number of missing values.

Table 57 SMT participants’ responses to 54 Likert-scale items

Item 1

strongly

disagree

2

disagree

3

agree

4 strongly

agree

%

missing

N % N % N % N % total

S71 1. I have a good understanding of my own and other people’s learning

0 0.0 0 0.0 13 52.0 12 48.0 25 0.0

S72 2. I keep my knowledge about teaching and learning and about my phase and/or subject specialism up to date

0 0.0 1 4.3 9 39.1 13 56.5 23 8.0

S73 3. I teach regularly and my lessons are rated good or outstanding

0 0.0 0 0.0 5 31.3 11 68.8 16 36.0

S74 4. I undertake regular and substantial professional development for myself

0 0.0 5 20.0 11 44.0 9 36.0 25 0.0

S75 5. I understand the differences between 0 0.0 0 0.0 9 36.0 16 64.0 25 0.0

101

good and outstanding lessons

S76 6. I regularly work with staff to improve their teaching practice

0 0.0 1 4.5 13 59.1 8 36.4 22 12.0

S77 7. I can design, monitor and assess programmes for effective teaching and learning

0 0.0 0 0.0 17 77.3 5 22.7 22 12.0

S78 8. I regularly participate in critical and reflective enquiry to develop teaching and learning across the Academy

1 4.2 4 16.7 15 62.5 4 16.7 24 4.0

S79 9. I understand what an Academy needs to do to become good and outstanding

0 0.0 0 0.0 12 48.0 13 52.0 25 0.0

S710 10. I regularly work with staff to help them improve their leadership and management

0 0.0 1 4.3 14 60.9 8 34.8 23 8.0

S711 11. I am an effective coach and use my coaching skills regularly to help others to improve their performance

0 0.0 0 0.0 18 78.3 5 21.7 23 8.0

S712 12. I regularly interact with staff in other schools, Academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our Academy

2 8.7 4 17.4 10 43.5 7 30.4 23 8.0

S713 13. I make regular and effective contributions to educational dialogue that shapes whole Academy policy and informs practice

0 0.0 0 0.0 13 54.2 11 45.8 24 4.0

S714 14. I contribute effectively to the critical development of Academy systems and structures in order to ensure quality teaching focused on student progress in learning

0 0.0 2 8.7 9 39.1 12 52.2 23 8.0

S715 15. The senior leadership team communicates a clear vision of where the Academy is going so that there is a shared sense of purpose

0 0.0 0 0.0 12 50.0 12 50.0 24 4.0

S716 16. Staff have a commitment to the whole Academy as well as to their department, key stage and/or year group

0 0.0 0 0.0 18 72.0 7 28.0 25 0.0

S717 17. The senior leadership team effectively promotes commitment among staff to the future development of the whole Academy

0 0.0 3 12.0 11 44.0 11 44.0 25 0.0

S718 18. There is effective communication between senior leaders and teachers

0 0.0 2 8.0 17 68.0 6 24.0 25 0.0

S719 19. There are processes for involving all staff in decision-making

0 0.0 2 8.0 16 64.0 7 28.0 25 0.0

S720 20. Teachers’ professional know-how is used in the formulation of Academy policy and goals

0 0.0 4 16.0 19 76.0 2 8.0 25 0.0

S721 21. Teachers’ professional know-how is used in the formulation of Academy policy, even where this leads to a questioning of established rules, procedures and practices

0 0.0 5 21.7 14 60.9 4 17.4 23 8.0

S722 22. Opportunities are provided for teachers to critically evaluate Academy policy

0 0.0 4 16.7 18 75.0 2 8.3 24 4.0

S723 23. Staff participate in important 0 0.0 5 20.0 17 68.0 3 12.0 25 0.0

102

decision-making

S724 24. There are processes for involving students in decision-making, including how best to improve classroom practice

0 0.0 5 20.8 12 50.0 7 29.2 24 4.0

S725 25. Staff have a good working knowledge of the Academy Development Plan

0 0.0 7 29.2 13 54.2 4 16.7 24 4.0

S726 26. Staff see the Academy Development Plan as relevant and useful to learning and teaching

0 0.0 6 25.0 16 66.7 2 8.3 24 4.0

S727 27. Staff development time is used effectively to realise Academy Development Plan priorities

0 0.0 1 4.2 19 79.2 4 16.7 24 4.0

S728 28. Staff development time is used effectively in the Academy, particularly through the explicit modelling of good practice

0 0.0 2 8.0 18 72.0 5 20.0 25 0.0

S729 29. The Academy provides support to allow staff joint planning time

0 0.0 6 25.0 17 70.8 1 4.2 24 4.0

S730 30. Teachers are encouraged to experiment with new ideas as a way of promoting professional growth and improving classroom practice

0 0.0 1 4.2 17 70.8 6 25.0 24 4.0

S731 31. Formal training provides opportunities for teachers to develop professionally

0 0.0 0 0.0 14 56.0 11 44.0 25 0.0

S732 32. Teachers are helped to develop skills to assess students’ work in ways that move their students on in their learning

0 0.0 1 4.2 18 75.0 5 20.8 24 4.0

S733 33. Teachers are helped to develop skills to observe learning as it happens in the classroom

0 0.0 1 4.2 18 75.0 5 20.8 24 4.0

S734 34. Senior leaders support teachers in sharing practice with other schools and Academies through networking

1 4.2 1 4.2 18 75.0 4 16.7 24 4.0

S735 35. Information is collected from teachers on those aspects of their work that they themselves think they do effectively

0 0.0 4 16.7 16 66.7 4 16.7 24 4.0

S736 36. Information is collected from teachers on effective ways they promote ‘learning to learn’ skills and knowledge among their students

0 0.0 6 25.0 17 70.8 1 4.2 24 4.0

S737 37. Information is collected from teachers on informal teacher networking in which they play an active role

0 0.0 9 39.1 14 60.9 0 0.0 23 8.0

S738 38. Teacher-initiated networking is an integral element of staff development

0 0.0 8 34.8 15 65.2 0 0.0 23 8.0

S739 39. ‘Learning how to learn’ is an issue discussed in staff development time

0 0.0 5 22.7 16 72.7 1 4.5 22 12.0

S740 40. There is a well-developed shared language and a rich dialogue about teaching and learning across the Academy

0 0.0 1 4.2 18 75.0 5 20.8 24 4.0

S741 41. We are good at systematic monitoring, use of data and feedback across the Academy

0 0.0 4 16.0 9 36.0 12 48.0 25 0.0

103

S742 42. We have efficient and effective support systems and structures across the Academy

0 0.0 0 0.0 13 52.0 12 48.0 25 0.0

S743 43. The senior leadership team are effective role models as ‘leaders of learning’,

0 0.0 1 4.0 14 56.0 10 40.0 25 0.0

S744 44. We have developed successful approaches to distributed leadership across the Academy where the pathways of accountability are working well

0 0.0 0 0.0 18 72.0 7 28.0 25 0.0

S745 45. The Academy is successful in helping parents/carers to support the learning of their children

0 0.0 9 36.0 16 64.0 0 0.0 25 0.0

S746 46. The governors play a full and appropriate part in strategic planning to improve teaching and learning and student outcomes

2 9.1 5 22.7 15 68.2 0 0.0 22 12.0

S747 47. The Academy involves a wide range of other stakeholders directly and powerfully in the education of students

0 0.0 2 8.3 18 75.0 4 16.7 24 4.0

S748 48. The Academy is good at securing the broadest possible impact of successful innovations in teaching and learning

0 0.0 2 8.3 21 87.5 1 4.2 24 4.0

S749 49. There is a widespread and well-established climate of trust in the Academy

0 0.0 1 4.0 20 80.0 4 16.0 25 0.0

S750 50. There is an appetite to face weaknesses and bring about changes for improvement in the Academy

0 0.0 1 4.2 14 58.3 9 37.5 24 4.0

S751 51. Good personal relationships and strong teams help us to face challenges with confidence

0 0.0 0 0.0 14 56.0 11 44.0 25 0.0

S752 52. The senior leadership team has the skills and resilience to successfully work through the more complex challenges that we are facing

0 0.0 0 0.0 8 32.0 17 68.0 25 0.0

S753 53. The senior leadership team is good at long term planning for capacity building

0 0.0 3 12.5 12 50.0 9 37.5 24 4.0

S754 54. As an Academy, we are good at celebrating success

0 0.0 1 4.2 16 66.7 7 29.2 24 4.0

LOWEST AND HIGHEST SCORES

Table 58 below lists the items with the five lowest and five highest mean responses. On average,

SMT respondents seemed to be less positive on the government fulfilling its role in developing

strategy to enhance teaching and learning leading to improvement of student outcomes (S746) and

the Academies being of assistance to parents/carers in their children’s learning (S745). Also the

respondents tended to disagree on information being collected from informal teacher networking

(S737), teacher initiated networking being used as an important platform for staff development

(S738) and specific time being allocated for staff jointly planning (S729). Interestingly, there was

tendency of agreement between SMT themselves, for example, knowing what needs to be done for

an Academy to be good and outstanding (S79), maintaining their professional development (S72) as

well as possessing skills and resilience to overcome challenges (S752) in addition to understanding

104

the difference between good and outstanding lessons (S755), and their lessons being rated as good

or outstanding (S73). These findings suggest that Academy SMT staff may perceive an unfriendly

macro working environment in terms of relevant governmental and Academy based policy

measures in supporting teaching and learning that focuses on student achievement, while they take

up the responsibility for their own professional development.

Table 58 Items of the five highest and least mean scores

N Min Max Mean SD

Items of the five lowest mean responses

S746 46. The governors play a full and appropriate part in strategic planning to improve teaching and learning and student outcomes

22 1 3 2.59 0.67

S737 37. Information is collected from teachers on informal teacher networking in which they play an active role

23 2 3 2.61 0.50

S745 45. The Academy is successful in helping parents/carers to support the learning of their children

25 2 3 2.64 0.49

S738 38. Teacher-initiated networking is an integral element of staff development

23 2 3 2.65 0.49

S729 29. The Academy provides support to allow staff joint planning time

24 2 4 2.79 0.51

Items of the five highest mean responses

S79 9. I understand what an Academy needs to do to become good and outstanding

25 3 4 3.52 0.51

S72 2. I keep my knowledge about teaching and learning and about my phase and/or subject specialism up to date

23 2 4 3.52 0.59

S75 5. I understand the differences between good and outstanding lessons

25 3 4 3.64 0.49

S752 52. The senior leadership team has the skills and resilience to successfully work through the more complex challenges that we are facing

25 3 4 3.68 0.48

S73 3. I teach regularly and my lessons are rated good or outstanding

16 3 4 3.69 0.48

(1=strongly disagree to 4=strongly agree)

Table 59 presents items with most and least consensus in the respondents’ responses ranging from

0.36 to 0.93 in terms of standard deviation, a measure employed to demonstrate the dispersion of

the SMT participants’ responses from their mean responses. In general, SMT staff reported the

least differences in whether or not they agree on their Academy being good at securing the impact

of innovation on teaching and learning (S748), effectively helping others’ improvement (S711),

being able to plan and evaluate effective teaching and learning programmes (S77), sound trust

existed within their Academy (S749) and staff development time well used to implement the

Academy development plan (S727). On the other hand, the least concurrences of SMT staff

responses were about their disagreement or agreement on regularly taking part in reflective

enquiry (S78), students being involved in the Academy’s decision making particularly relating to

classroom practice (S724), regularly attending professional development activities (S74), being good

at using evidence based data (S741) and frequently networking outside the Academy (S712) for

improvement.

105

These findings imply the issues of fairness and democracy across Academies in relation to

opportunity being created for improvement of teaching and learning through self-reflection,

participation in relevant professional development activities, use and analysis of data, engagement

in external partnerships and involvement of students in decision making.

Table 59 Items of the five most and five least consensuses in responses (ascending rank by standard deviation (SD)

N Min Max Mean SD

Items of the five most consensuses in responses

S748 48. The Academy is good at securing the broadest possible impact of successful innovations in teaching and learning

24 2 4 2.96 0.36

S711 11. I am an effective coach and use my coaching skills regularly to help others to improve their performance

23 3 4 3.22 0.42

S77 7. I can design, monitor and assess programmes for effective teaching and learning

22 3 4 3.23 0.43

S749 49. There is a widespread and well-established climate of trust in the Academy

25 2 4 3.12 0.44

S727 27. Staff development time is used effectively to realise Academy Development Plan priorities

24 2 4 3.13 0.45

Items of the five least consensuses in responses

S78 8. I regularly participate in critical and reflective enquiry to develop teaching and learning across the Academy

24 1 4 2.92 0.72

S724 24. There are processes for involving students in decision-making, including how best to improve classroom practice

24 2 4 3.08 0.72

S74 4. I undertake regular and substantial professional development for myself

25 2 4 3.16 0.75

S741 41. We are good at systematic monitoring, use of data and feedback across the Academy

25 2 4 3.32 0.75

S712 12. I regularly interact with staff in other schools, Academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our Academy

23 1 4 2.96 0.93

(1=strongly disagree to 4=strongly agree)

EXPLORING PATTERNS IN SENIOR LEADER PARTICIPANTS’ RESPONSES

In combination with varimax rotation to hypothesise potential components being uncorrelated to

each other, Principal Components Analysis (PCA) was performed to explore possible underlying

patterns in SMT participants’ responses to the Likert-scale items by reducing the number of items

and dimensionality. As compared to the number of items in the questionnaire, the sample size

obtained in the study was relatively small. Hence, missing values of individual items were replaced

with the mean during the procedure of principal component analysis. However, the item “I teach

regularly and my lessons are rated good or outstanding” (S73) was excluded from the analysis

because nearly one-third of the participants reported with missing responses.

Multiple criteria were taken into account to determine and retain items/components for

interpretation including i) an eigenvalue greater than 1.0; ii) the scree test; iii) proportion of

variance accounted for; and iv) interpretability (e.g., item loadings greater than 0.4 (and on only

106

one given component to be expected), items of a given component sharing similar conceptual

meaning. PCA was repeatedly conducted. After removing 33 items, four potentially meaningful

components with eigenvalues greater than 1 emerged that explained 73.6% of the total variance.

All the remaining 20 items had communalities greater than 0.5 (rounded). Bartlett’s test was

statistically significant at 0.00 level and the value of overall Kaiser Meyer Olker measure of sampling

adequacy was 0.55 indicating a mild satisfactory analysis.

Table 60 shows the rotated component matrix for the items with the estimated loadings. These

factors were found to be reliable with Cronbach's Alpha value ranging from 0.84 to 0.88. Since this

study is exploratory in nature some items with loadings of greater than 0.400 on more than two

components were retained for the purpose of identifying underlying dimensions to be tested in a

larger sample size in a further study.

Table 60 Rotated Component Matrix Strand Seven

Item Component

1 2 3 4

S734 34. Senior leaders support teachers in sharing practice with other schools and Academies through networking

.890

S749 49. There is a widespread and well-established climate of trust in the Academy

.771

S712 12. I regularly interact with staff in other schools, Academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our Academy

.764

S714 14. I contribute effectively to the critical development of Academy systems and structures in order to ensure quality teaching focused on student progress in learning

.648 .503

S746 46. The governors play a full and appropriate part in strategic planning to improve teaching and learning and student outcomes

.636

S713 13. I make regular and effective contributions to educational dialogue that shapes whole Academy policy and informs practice

.573 .471

S78 8. I regularly participate in critical and reflective enquiry to develop teaching and learning across the Academy

.459 .455 .448

S727 27. Staff development time is used effectively to realise Academy Development Plan priorities

.813

S750 50. There is an appetite to face weaknesses and bring about changes for improvement in the Academy

.788

S728 28. Staff development time is used effectively in the Academy, particularly through the explicit modelling of good practice

.781

S741 41. We are good at systematic monitoring, use of data and feedback across the Academy

.711

S753 53. The senior leadership team is good at long term planning for capacity building

.686 .565

S742 42. We have efficient and effective support systems and structures across the Academy

.423 .634

S726 26. Staff see the Academy Development Plan as relevant and useful to learning and teaching

.872

107

S725 25. Staff have a good working knowledge of the Academy Development Plan .803

S723 23. Staff participate in important decision-making .625

S720 20. Teachers’ professional know-how is used in the formulation of Academy policy and goals

.866

S719 19. There are processes for involving all staff in decision-making .838

S721 21. Teachers’ professional know-how is used in the formulation of Academy policy, even where this leads to a questioning of established rules, procedures and practices

.659

S718 18. There is effective communication between senior leaders and teachers .528

Reliability (alpha) 0.88 0.88 0.85 0.84

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 6 iterations.

FOUR UNDERLYING THEMES

The four underlying themes are outlined in Table 61 below.

Table 61 Underlying Themes for staff questionnaire

Theme one explains 41.3% of the variance with a

reliability of Cronbach’s Alpha value of 0.88 including

seven items

I engage in learning and dialogue about the leadership of our Academy with a wide range of colleagues in a positive climate of trust

Theme two has a Cronbach’s Alpha value of 0.88

explaining 13.7% of the variance and consists of six items

We monitor progress carefully, face up to challenges and systematically support and encourage teacher progress

Theme three with a reliability of Cronbach’s Alpha value of

0.85 explaining 10.6% of the variance contains three items

The Academy Development Plan is relevant to learning and teaching and is a focus for shared decision making

Theme four has a Cronbach’s Alph value of 0.84 explaining

6.5% of the variance and includes four items

All staff are involved in Academy decision making

COMPUTING NEW VARIABLES

Because the exploratory factor analysis reveals four clear themes, for this Strand, new variables

were computed for each individual and these were then used for the Perimeta model.

SUMMARY AND FINDINGS FROM STRAND SEVEN

This Strand explored senior leaders’ perceptions from all three Academies and how they are viewed

to be fulfilling the OCL Charter.

The findings from the 55 item questionnaire seems to suggest that senior management teams

(SMT) perceive an unfriendly macro working environment in terms of relevant governmental and

Academy based policy measures in supporting teaching and learning that focuses on student

108

achievement. However, SMT seem to demonstrate a confidence in taking up the responsibility for

their professional development.

The findings also imply the issues of fairness and democracy across Academies in relation to

opportunity being created for improvement of teaching and learning through self-reflection,

participation in relevant professional development activities, use and analysis of data, engagement

in external partnerships and involvement of students in decision making.

Principal Components Analysis (PCA) was performed to explore possible underlying patterns in SMT

participants’ responses to the Likert-scale items by reducing the number of items and

dimensionality. Four underlying themes were revealed which were used to reduce and strengthen

the data for further analysis (by Perimeta).

STRAND EIGHT & NINE – CONTEXT FOR STUDENT LEARNING - KEY PERFORMANCE

INDICATORS & KEY OUTCOME FOR YEAR 11 STUDENTS – GCSE/EXAM RESULTS

Data from Key Performance Indicators, such as attendance, free school meals (FSM, exclusions,

special educational needs (SEND), gender and GCSE results.

SAMPLE AND DATA COLLECTION

KPI data and the GCSE results from Academy 2 (153 records) and Academy 3 (166 records) were

explored for differences between and within Academies with regards to gender, FSM and SEN in

relation to GCSE results in English, Maths and Science.

DIFFERENCES WITHIN ACADEMIES

Academy 2

Note that only the results of i) GCSE Mathematics, ii) GCSE English Language & GCSE English

Literature, and iii) GCSE Science Double Award/GCSE Biology & GCSCE Chemistry & GCSE Physics

that were taken by the majority of students were analysed by gender, FSM and SEN. Table 54

indicates the number of students by gender, FSM and SEN where 55.6% of students were male.

There were 20.3% of students entitled to FSM. Notably, 47.7% of students have received the

support of either SEN school action or SEN school action plus.

Table 62 Frequency of Academy 2 students by gender, FSM and SEN

Students by gender

Students by FSM

Students by SEN

109

Gender Frequency %

male 85 55.6

female 68 44.4

Total 153 100.0

FSM Frequency %

not

Eligible

122 79.7

Eligible 31 20.3

Total 153 100.0

SEN Frequency %

No 80 52.3

School

action

38 24.8

School

action

plus

35 22.9

Total 153 100.0

GCSE Mathematics: All 153 students at Academy 2 took GCSE Maths. One student (0.7%) was

graded as ‘X’ for completing only part of the course. Over half the students (83, 54.2%) received a C

grade (see Table 63). Students graded at either AA to B totaled 29, (19%) and D to F totaled 26,

(17%).

Table 63 Academy 2 Frequency of GCSE Maths grades

Grade Frequency % %

AA 6 3.9

19.0 A 9 5.9

B 14 9.2

C 83 54.2 54.2

D 9 5.9

17.0 E 6 3.9

F 11 7.2

G 12 7.8 7.8

U 2 1.3 1.3

110

X 1 0.7 0.7

Total 153 100.0 100.0

GCSE Mathematics by Gender: Of the 153 students in Academy 2 who took GCSE Maths, 85

(55.6%) were male. As shown on the chart below (Figure 9), more male than female students were

awarded grades A to C. A higher percentage of female students were among the D to G grades. The

results reflect the existing pattern of Maths attainment between the genders where male students

generally outperform female students.

Table 64 GCSE Maths results by gender

GCSE Mathematics by FSM: Of 153 students in Academy 2 who took the GCSE Maths exam, 31

(20.3%) were FSM eligible students. Table 65 below shows that a large percentage of FSM and Non

FSM students achieved grade C. However, there was a marked difference in the percentage of FSM

students at 22.6% achieving grade G as compared with 4.1% of Non FSM students.

Table65 GCSE Maths results by FSM

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

Ge

nd

er

GCSE Mathematics (sch_2)

Male

Female

-10.0%

10.0%

30.0%

50.0%

70.0%

AA A B C D E F G UPer

cen

tage

wit

hin

FS

M

GCSE Mathematics (Sch_2)

No…

111

GCSE Mathematics by SEN: There were a high proportion of students in Academy 2 identified as

having learning difficulties or disabilities. Of 153 students in Academy 2 who took GCSE Maths, 38

(24.8%) received specific additional help (i.e. SEN status = school action), and 35 (22.9%) who

received further external help (i.e. SEN status = action_plus). As expected, and shown in Figure 11

below, a greater percentage of No_SEN students were rewarded AA to C grades than their Yes_SEN

counterparts. Additionally, a much higher percentages of students with action_SEN (5.3% / 10.5%)

or action_plus_SEN (22.9% / 20.0%) than those students without SEN (1.3% / 1.3%) had a grade of

F/G. Noticeably, there were 2 (5.3%) action_SEN students that had a grade of A indicating some

impact of the interventions made by the Academy for those SEN students. However, these two

students were labelled as gifted or talented.

Table 66 GCSE Mathematics by SEN

GCSE Mathematics by EAL: Only 12 (7.8%) students reported English as an Additional Language.

GCSE Mathematics by Ethnicity: Of the 153 students, 135 or 88.2% were white English.

GCSE English/GCSE English Language & GCSE English Literature: Students can take GCSE English or

both GCSE English Language and GCSE English Literature. In Academy 2, there were 80 students

(52.3%) who took GCSE English while 71 students (46.4%) sat both English Language and English

Literature exams (note that two students (1.3%) only took GCSE English Language but not English

Literature). Table 67 shows the percentage of students taking each of the GCSE English exams and

those who did not.

Table 67 Academy 2 Frequency of GCSE English / English Language / English Literature results

GCSE English

GCSE English Language

GCSE English Literature

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

SEN

GCSE Mathematics (Sch_2)

No_SEN

action_SEN

action_plus_SEN

112

Grade Frequency %

NoSitIn 73 47.7

C 7 4.6

D 38 24.8

E 21 13.7

F 11 7.2

G 2 1.3

U 1 .7

Total 153 100.0

Grade Frequency %

NoSitIn 80 52.3

AA 1 .7

A 5 3.3

B 12 7.8

C 46 30.1

D 9 5.9

Total 153 100.0

Grade Frequency %

NoSitIn 82 53.6

A 2 1.3

B 17 11.1

C 18 11.8

D 27 17.6

E 7 4.6

Total 153 100.0

Moreover, among those who took both English Language and English Literature, a total of 47.8% of

students achieved the same grade for both the GCSEs (see Table 68 below). The remaining majority

of students, however, scored a higher grade in English Language than in English Literature, where

31% of students achieved grade at C in English Language and grade of D in English Literature.

Table 68 GCSE English Language and English Literature results

113

GCSE English/GCSE English Language and GCSE English Literature by Gender: Among those

students that took GCSE English, no students achieved grade B or higher (see Table 69 below). More

females than males achieved grades C, E, F and G. However, more males than females achieved a

grade D - 54.2% compared to the females at 31.6%.

Table 69 GCSE English by Gender

Interestingly, and on the contrary, those students who took GCSE English Language all had a grade

of D or above, where the majority scored grade C, 64.8% for male and 61.1% for female (see Table

70 below). The percentages of male and female students with a grade of A or B or C were similar;

but there were more females than male students who were awarded a D grade in the GCSE.

Table 70 GCSE English Language by Gender

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

AA A B C D E F G U

Per

cen

tage

wit

hin

Ge

nd

er

GCSE English (Sch_2)

Male

Female

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

Ge

nd

er

GCSE English Language (Sch_2)

Male

Female

114

As to those students who took GCSE English Literature, in general, slightly higher percentages of

male students than those of female students achieved a B, D or E grade. However, the percentage

of female student with a grade C was twice as high as those of male students.

Table 71 GCSE English Literature by Gender

GCSE English/GCSE English Language and GCSE English Literature by FSM: There was little

difference between FSM and non-FSM at grade C in GCSE English. However, the number of FSM

students achieving grade D was half that of non-FSM. The percentage of FSM students achieving

grades E, F or G was greater than non-FSM students.

Table 72 GCSE English by FSM

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

Ge

nd

er

GCSE English Literature (Sch_2)

Male

Female

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

AA A B C D E F G U

Per

cen

tage

wit

hin

FSM

GCSE English (Sch_2)

No_FSM

115

In GCSE English Language, no FSM students achieved grade B or higher compared to 28.2% of non-

FSM students who achieved grade B or higher. Although at grade C there was little difference

between non-FSM and FSM students. However, the difference at grade D was more marked where

three times more FSM students achieved grade D compared to non-FSM students.

Table 73 GCSE English Language by FSM

Similarly, with regarding to GCSE English Literature, a lower percentage of FSM students achieved

grade C or higher compared to non-FSM. In contrast, there was a marked difference in the

percentage of FSM students achieving the grades D or E (see Table 74 below) compared with non-

FSM.

Table 74 GCSE English Literature by FSM

GCSE English/GCSE English Language and GCSE English Literature by SEN: The percentage of No

SEN students achieving grades C or D in GCSE English were greater than those students identified as

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

FSM

GCSE English Language (Sch_2)

No_FSM

Yes_FSM

-10.0%

10.0%

30.0%

50.0%

70.0%

AA A B C D E F G UPer

cen

tage

wit

hin

FS

M

GCSE English Literature (Sch_2)

No_F…

116

SEN. Also, a relatively higher percentage of action plus SEN students achieved an E or F grade than

action SEN students.

Table 75 GCSE English Literature by SEN

Only students identified as No SEN achieved grade B or above in GCSE English Language (see Table

76 below) with the majority of SEN students achieving a grade C.

Table 76 GCSE English Language by SEN

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

AA A B C D E F G U

Per

cen

tage

wit

hin

SEN

GCSE English (Sch_2)

No_SEN

action_SEN

action_plus_SEN

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

SEN

GCSE English Language (Sch_2)

No_SEN

action_SEN

action_plus_SEN

117

As shown in Table 77 below, No action plus SEN students scored grade B or above in GCSE English

Literature. However, the differences between the groups at grade C and D were less marked and a

greater percentage of action plus SEN than action only SEN achieved grade C.

Table 77 GCSE English Literature by SEN

GCSE Science Double Award/GCSE Biology and GCSE Chemistry and GCSE Physics: Students can

opt to take the GCSE Science Double Award or three separate sciences - GCSE Biology, Chemistry or

Physics. In Academy 2, 20 (13.1%) did not take any of the science exams, while 108 (70.6%)

students took the GCSE Science Double Award and 25 (16.3%) students took the three science

exams. Table 78 shows the percentage of students sitting either the Science Double Award, the

separate sciences as well as those not taking either.

Table 78 Academy 2 Frequency of GCSE Science Double Award / Biology / Chemistry / Physics results

GCSE Science Double

Award

Grade Frequency %

NoSitIn 45 29.4

AA 2 1.3

GCSE Biology

Grade Frequency %

NoSitIn 128 83.7

AA 1 .7

A 9 5.9

GCSE Chemistry

Grade Frequency %

NoSitIn 128 83.7

A 5 3.3

AA 3 2.0

GCSE Physics

Grade Frequency %

NoSitIn 128 83.7

A 3 2.0

AA 1 .7

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

SEN

GCSE English Literature (Sch_2)

No_SEN

action_SEN

action_plus_SEN

118

AB 1 .7

BB 5 3.3

BC 16 10.5

CC 45 29.4

CD 10 6.5

DD 7 4.6

DE 5 3.3

EE 7 4.6

EF 2 1.3

FF 1 .7

FG 1 .7

GG 3 2.0

U 3 2.0

Total 153 100.0

B 6 3.9

C 6 3.9

D 3 2.0

Total 153 100.0

B 8 5.2

C 5 3.3

D 4 2.6

Total 153 100.0

B 10 6.5

C 8 5.2

D 3 2.0

Total 153 100.0

GCSE Science Double Award/GCSE Biology and GCSE Chemistry and GCSE Physics by Gender:

There were 46 female and 62 male students taking the GCSE Science Double Award exam with a

slightly higher percentage (72.9%) of male than female students (67.6%). A higher percentage of

female than male students (76.1% and 54.8%, respectively) achieved grade CC or above.

119

Table 79 GCSE Double Science Award by gender

With regard to GCSE Biology results, whilst no female students achieved grade AA (nor B grade), a

considerable higher percentages of females achieved grade A than male students.

Table 80 GCSE Biology results by gender

The results for GCSE Chemistry were similar for female and male students at grade AA or B (see

Table 81 below). However, relatively lower percentages of females achieved grade A or grade D in

contrast to a higher percentages of females achieving grade C.

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA AB BB BC CC CD DD DE EE EF FF FG GG U

Per

cen

tage

wit

hin

Ge

nd

er

GCSE Science Double Award (Sch_2)

Male

Female

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G UPer

cen

tage

wit

hin

Ge

nd

er

GCSE Biology (sch_2)

Male

Female

120

Table 81 GCSE Chemistry results by gender

In general, there was little difference between female and male students at grade AA or B the in

GCSE Physics exam.

Table 82 GCSE Physics results by gender

GCSE Science Double Award/GCSE Biology and GCSE Chemistry and GCSE Physics by FSM: Of the

108 students who took the GCSE Science Double Award exam, 21 were FSM entitled students. A

higher percentage of non-FSM students than FSM students had a double grade at CD or above

(78.2% and 52.4% respectively). In contrast, the percentage of FSM students was three times as

much as the non-FSM who achieved a double grade of DD or below or ungraded (33.3% and 11.5%

respectively).

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G UPer

cen

tage

wit

hin

Ge

nd

er

GCSE Chemistry (sch_2)

Male

Female

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

Ge

nd

er

GCSE Physics (sch_2)

Male

Female

121

Table 83 GCSE Double Science Award by FSM

Among the 25 students that took the three science-related subject exams, 2 were FSM entitled

both achieving grade C in GCSE Biology as well as in GCSE Physics; however, one achieved grade C

and the other grade D in the GCSE Chemistry exam. The remaining 23 non-FSM students achieved

grade B or higher in GCSE Biology, GCSE Chemistry and GCSE Physics respectively.

Table 84 GCSE Biology results by FSM

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA AB BB BC CC CD DD DE EE EF FF FG GG U

Per

cen

tage

wit

hin

FSM

GCSE Science Double Award (Sch_2)

No_FSM

10

122

Table 85 GCSE Chemistry results by FSM

Table 86 GCSE Physics results by FSM

GCSE Science Double Award/GCSE Biology and GCSE Chemistry and GCSE Physics by SEN: Of the

108 students who took the GCSE Science Double Award exam, 28 were defined as SEN students.

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

FSM

GCSE Chemistry (Sch_2)

No_FSM

Yes_FSM

100%

123

Table 87 GCSE Science Double Award results by SEN

Table 88 GCSE Biology res ults by SEN

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA AB BB BC CC CD DD DE EE EF FF FG GG U

Per

cen

tage

wit

hin

SEN

GCSE Science Double Award (Sch_2)

No_SEN

action_SEN

action_plus_SEN

100%

100%

124

Table 89 GCSE Chemistry results by SEN

Table 90 GCSE Physics results by SEN

100%

100%

100% 100%

125

Academy 3

Only the results of i) uncapped average point score (APS) / capped average point score (APS) / total

points, ii) [GCSE] English, iii) [GCSE] Maths taken by the majority of students were analysed by

gender, FSM and SEN.

Table 91 summarises student composition by gender, FSM and SEN in Academy 3. Overall, 53.6% of

students were male, which is similar to Academy 2 (see Table 75.). There were 34.3% of students

entitled to FSM, and in Academy 2 only 20.3% of students entitled to FSM. Considerable 54.2% of

students, a 6.5 percentage points higher than that of in Academy 2, have received the support of

either SEN school action or SEN school action plus or been identified as statement of SEN.

Table 91 Academy 3 summary of students by gender, FSM and SEN

Students by gender

Gender Frequency %

male 89 53.6

female 77 46.4

Total 166 100.0

Students by FSM

FSM Frequency %

not

Eligible

109 65.7

Eligible 57 34.3

Total 166 100.0

Students by SEN

SEN Frequency %

No 75 45.2

School

action

48 28.9

School

action plus

34 20.5

Statement

of SEN

9 5.4

Total 166 100.0

Uncapped average point score (APS) and capped average point score (APS): Note: After cross

checking between capped APS and total points as well as records of individual exams provided in

the dataset, it is unclear whether the uncapped APS, capped APS and total points of a pupil

provided in the dataset were calculated based on i) average total point scores of all of the pupil

results at KS4, ii) average total point scores of the pupil’s best 8 results including English and Maths

at KS 4, and iii) total point scores of all of the pupil results at KS4, respectively. Take the minimum

capped APS (see Table 77.) as an example, which were 23.3 suggesting a minimum total points of

186.4 obtained from multiplying 23.3 by 8. However, as shown in Table 77., the minimum total

points were 35.0.

126

Eventually, in addition to the English and Maths exam results, only capped APS was analysed and

reported below.

Table 92 Mean and SD of uncapped and capped average point score (APS)

N Minimum Maximum Mean SD

Uncapped average point score (APS) 164 23.3 53.2 40.1 6.0

Capped average point score (APS) 164 23.3 58.0 43.5 6.8

Total Points 164 35.0 876.0 509.7 167.3

Capped Average Point Score by Gender: An independent t-test was carried to explore whether

there was difference between genders in mean in average point score (APS). The results showed

that there was statistically significant difference in mean APS between genders at α = 0.05 level. On

average males have a lower mean in capped APS (42.22 versus 44.98 in mean) than female

students. However this varied much less around the mean (5.27 versus 8.09 in SD) (see Table 93).

Table 93 Mean and SD of capped APS between gender

Gender N Mean SD

Male 88 42.22 5.27

Female 76 44.98 8.09

t(162)=-2.62, p < 0.05

Capped Average Point Score by FSM: An independent t-test was conducted to compare the mean

APS scores of non-FSM and FSM student groups. There was a statistically significant difference in

mean APS between non-FSM and FSM student groups at α = 0.05 level. On average students who

were not entitled to FSM scored about 3.47 APS higher than students who were entitled to FSM

(44.68 versus 41.21 in mean) (see Table 94).

127

Table 94 Mean and SD of capped APS between non-FSM and FSM student groups

FSM N Mean SD

No_FSM 108 44.68 6.70

Yes_FSM 56 41.21 6.59

t(162)=3.47, p < 0.01

Capped Average Point Score by SEN: An independent t-test was carried to compare the mean APSs

of non-SEN and SEN student groups. There was a statistically significant difference in mean APS

between non-SEN and SEN student groups at α = 0.05 level. The average mean of 6.38 of APS

scored more by non-SEN (46.96 in mean) than SEN (40.58 in mean) groups (see Table 95).

Table 95 Mean and SD of capped APS between No_SEN and Yes_SEN student groups

SEN N Mean SD

No_SEN 75 46.96 6.49

Yes_SEN 89 40.58 5.71

t(162)=6.70, p < 0.01

English: Only 3% of students did not the take English exam (see Table 96).

Table 96 Frequency of English results of

Academy 3 students

Grade Frequency %

NoSitIn 5 3.0

A 3 1.8

B 20 12.0

128

C 54 32.5

D 26 15.7

E 33 19.9

F 20 12.0

G 5 3.0

Total 166 100.0

GCSE English by Gender: A small proportion (3.4%) of male, but no female students achieved grade

A. As shown in Figure below, a greater percentage of male than female students achieved grade C

or lower. In contrast, there was considerably higher percentage of female students who achieved

grade B than male students. Further, it is not clear whether English exams taken by Academy 2 and

Academy 3 were the same GCSE exam. However, based on both scoring systems provided (which

were ‘grading’), it seems that a greater percentage of Academy 3 than Academy 2 students

achieved grade C or above.

Table 97 Academy 3 English results

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

Ge

nd

er

English (sch_3)

Male

Female

129

English by FSM: Among the students that took the English exam, a higher percentages of non-FSM

than FSM students achieved grade A, B or C. In contrast, more FSM than non-FSM students

achieved grade E, F, or G.

Table 98 Academy 3 English results by FSM

Maths: In Academy 3, 97.6% students took the Maths exam (see Table 99).

Table 99 Academy 3 Frequency of grades

Grade Frequency %

NoSitIn 4 2.4

A* 3 1.8

A 11 6.6

B 31 18.7

C 58 34.9

D 12 7.2

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA A B C D E F G U

Per

cen

tage

wit

hin

FSM

English (Sch_3)

No_FSM

Yes_FSM

130

E 10 6.0

F 16 9.6

G 21 12.7

Total 166 100.0

Maths by Gender

Table 100 Academy 3 Maths results

Maths by FSM

Table 101 Academy 3 Maths results by FSM

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

A* A B C D E F G U

Per

cen

tage

wit

hin

Ge

nd

er

Maths (sch_3)

Male

Female

.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

A* A B C D E F G U

Per

cen

tage

wit

hin

FSM

Maths (Sch_3)

No_FSM

Yes_FSM

131

DIFFERENCES BETWEEN ACADEMIES

Note that only the results of English and Maths exams were compared between the two

Academies.

Table 102 Academy 2 and 3 GCSE English/English results

Table 103 Academy 2 and 3 GCSE Maths/Maths results

SUMMARY AND FINDINGS FROM STRANDS EIGHT & NINE

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

AA/A* A B C D E F G U

Per

cen

tage

GCSE English / English

Sch_2

Sch_3

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

AA/A* A B C D E F G U

Per

cen

tage

GCSE Mathematics / Maths

Sch_2

Sch_3

132

These Strands used data from Key Performance Indicators, such as attendance, free school meals

(FSM, exclusions, special educational needs (SEND), gender and GCSE from Academy 2 and

Academy 3 for differences between and within Academies with regards to gender, FSM and SEN in

relation to GCSE results in English, Maths and Science.

Generally, in Academy 2, male students performed poorer in GCSE English and Double Science but

outperformed female students in terms of percentages at Grade C or above in Maths. A similar

trend applied to Academy 3 male and female students in GCSE English and Math exams. However,

overall in Academy 3, female students had higher mean scores in Capped APS than male students.

In both Academies, disadvantaged (ie FSM students) and SEN students were underperforming than

their counterpart students.

Comparing the two Academies in English and Maths exam results, Academy 2 students had

performed slightly better in Maths but underperformed in English in comparison to Academy 3

students. (Note: it needs to confirm that the marking systems used in those English Exams taken by

the two Academy students were the same.) Regarding gender difference, a higher percentage of

male students than female students in Academy 2, but similar percentages of male and female

students in Academy 3 received Grade A- C in the Maths exam.

133

134

HIEARCHICAL PROCESS MODEL

Hierarchical Process Modelling (HPM) was used in addition to the traditional social science methods

of data analysis to each of the Strands, using the Perimeta software as described earlier. The

findings are presented below beginning with a summary of the findings for that Strand from the

social science methods.

STRAND ONE - DEVELOPMENT OF STUDENTS AS LEARNERS IN YEAR 7

SUMMARY FROM SOCIAL SCIENCE DATA ANALYSIS

Strand One was designed to assess Year 7 students’ self-reporting of their learning dispositions,

using the Effective Lifelong Learning Inventory's seven scales.

The findings from the social science method demonstrated there was a significant increase in mean

score between Time 1 and Time 2 on six of the seven dimensions of learning power, based on the

Effective Lifelong Learning Inventory, which means that something other than chance must account

for this increase. The increase was not uniform across all three Academies where Academy 1 and 3

were more similar in terms of their statistical significance in many of the dimensions of learning.

This seems to suggest that whilst there are significant increases in scores for all the Academies

overall, except for Resilience, there are other factors which contributed to the differing levels of

significance in the seven dimensions between Academies. This could be due to a range of factors

such as the students themselves, the delivery of the interventions or the overall leadership of the

Academy and are worth exploring further.

PERIMETA ANALYSIS

A total of 1918 data points (for 274 respondents x 7 statements) were entered in the Strand 1 Perimeta model and verified through statistical comparison with the source data.

Performance ratings representing evidence of self-perception of learning power were calculated

from the performance scores assuming a relatively high level of confidence of 75% in the responses,

on the basis that ELLI is a research-validated questionnaire.

The Perimeta model of Strand 1 performance was used to propagate the evidence of Responses

and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender

and by Question). For ease of computation the Strand 1 model was configured in four parts, for

Male and Female students and Times 1 and 2.

135

Table 145 presents a full list of results for Strand 1 covering performance Overall, by Gender, by

Academy and by Question. Results for male and female students are tabulated separately.

Table 104 presents dashboard summaries of the full results of the Perimeta analysis of the

experience of male and female students at the beginning (Time 1) and end (Time 2) of Year 7.

Table 104 Strand 1 - Results dashboards

Male students, Time 1 Male students, Time 2

Female students, Time 1 Female students, Time 2

Strand 1 performance analysis by Question

Table 105 illustrates the distribution of all responses at both Time 1 and Time 2 for Strand 1

according to the strength of evidence of self-perception of learning power. The horizontal axis

represents positive evidence (POS) and the vertical axis represents negative evidence (NEG), so that

the more uncertain (UNC) responses would appear towards the lower left of the graph. This

indicates both the very wide range of strengths of perception in dimensions of learning (10 to 80

percent positive) and the relatively steady degree of uncertainty (20 to 30 percent). The results

show a bias towards positive perceptions, with average positive evidence (57%) outweighing

average uncertainty (27%) and average negative evidence (16%).

136

Table 105 Strand 1 – Distribution of responses

The strongest responses in terms of strength of evidence in the statements are identified in Table

106.

Table 106 Strand 1 – Strongest responses

Gender Strongest positive evidence Strongest negative evidence

Time 1 Time 2 Time 1 Time 2

Male students S1 6 Meaning Making

67%

(+ 2 over 50%)

S1 7 Critical Curiosity

80%

(+ 5 over 50%)

S1 5 Creativity 29%

(+ 2 over 25%)

S1 4 Resilience 26%

(+ no others > 25%)

Female students S1 1 Changing &

Learning 82%

(+2 others > 50%)

S1 1 Changing &

Learning 85%

(+5 others > 50%)

S1 4 Resilience 50%

(+ no others > 25%)

S1 4 Resilience 62%

(+ no others > 25%)

Table 107 summarises the change in student perceptions between the start and end of Year 7. This

shows that students’ positive self-perceptions of their learning power increased (and/or negative

perceptions decreased) during Year 7 for five of the seven ELLI dimensions, but did not change for

Changing & Learning and actually fell for Resilience. The generally modest changes in uncertainty

mirrored this pattern, decreasing or remaining the same over the year but increasing for Resilience.

137

Table 107 Strand 1 – Changes in performance over Year 7 by Question and Gender

Q Statement Males Females

POS UNC NEG POS UNC NEG

All All + -- -- ++ -- --

S1 1 Changing & Learning l l l l l l

S1 2 Learning Relationships + l . + - -

S1 3 Strategic Awareness ++ - -- + l -

S1 4 Resilience - + l - - +

S1 5 Creativity ++ - - - ++ - - - -

S1 6 Meaning Making + - - ++ - - - -

S1 7 Critical Curiosity + - -- ++ - - - -

Key: ++ substantial increase; + increase; l similar; – decrease; - - substantial decrease

Strand 1 performance analysis by Gender

Differences between the perceptions of male and female students are also evident in Table 66. In

general female students were more positive than males that their learning power increased (and

less negative). With regard to Resilience, where both genders perceive a worsening of learning

power, females were less uncertain while males are more uncertain, and females were more

negative while males were unchanged.

Strand 1 performance analysis by Academy

Performance by Academy differed substantially at Time 1 in terms of evidence of self-perception of

learning power, with Academy 3 (53% positive for males and 44% for females) outperforming

Academy 1 (32% and 36%) and Academy 2 (24% and 26%). At Time 2 the Academies are much

closer in performance but with Academy 1 on top (87% and 76%) of Academy 3 (80% and 80%) and

Academy 2 (80% and 63%). Therefore there were substantial improvements in positive self-

perceptions of learning power at each Academy, with the most marked improvements at Academy

1 (55% and 40%) and Academy 2 (56% and 37%) ahead of Academy 3 (27% and 36%).

Results obtained using the Perimeta model of Strand 1 are compared in Table 108 with the results

of standard statistical tests provided by the ECHO project team.

Table 108 Strand 1 – Comparison of results from Perimeta and statistical tests

Perimeta results Statistical test results Notes

Positive self-perception of learning

power:

increased for five dimensions;

did not change for Changing &

Learning;

decreased for Resilience

Learning power (using T-test):

mean scores increased significantly

for six dimensions;

decreased for Resilience

Perimeta results broadly consistent

with T-test means

Uncertainty in learning power

decreased or remained the same

for six dimensions

increased for Resilience

Standard deviation of scores

decreased a little or no change

Perimeta uncertainties and

statistical standard deviations not

related

Positive self-perception of learning Performance by Academy (using Perimeta results apparently the

reverse of the ANOVA results,

138

power by Academy

substantial differences between

Academies at Time 1

much less difference at Time 2

ANOVA):

no statistically significant

differences between

Academies at Time 1

significant differences at Time 2.

revealing deeper insight into

positive perceptions

Academy 1 improved the most out

of the three Academies

Academy 1 improved the most out

of the three Academies

Full agreement between Perimeta

and social science analyses

STRAND TWO - STUDENT PERCEPTIONS OF THE IMPACT OF THE EDUCATION CHARTER

SUMMARY FROM SOCIAL SCIENCE DATA ANALYSIS

For Strand Two the questionnaire was designed to assess students' perceptions about their

learning, teaching, progress and achievement in an Oasis Academy.

The findings analysed with traditional social science methods showed there to be no mean

differences in agreement or disagreement for 10 of the 42 items assessing student perceptions on

their learning, teaching, progress and achievement in an Oasis Academy. However, Academy 1

student respondents reported averagely higher mean responses than their counterparts of

Academy 3 to the remaining 32 items which suggests that, to some degree, students in Academy 1

tend to agree more than Academy 3 students.

With regards to gender differences, the statistical results indicated that male and female student

respondents on average provided similar responses to 20 out of 42 items, although, across the

Academies female student respondents reported significantly higher means than male student

respondents on the remaining 22 items. Thus, there appear to be gender differences on over half

of the items, possibly suggesting a difference in how males and females perceive at least some of

their learning, teaching, progress and achievement.

PERIMETA ANALYSIS

A total of 15792 data points (for 376 respondents x 42 statements) were entered in the Perimeta model for Strand 1 and verified through statistical comparison with the source data.

Performance ratings representing evidence of strength of self-perceptions of learning were

calculated from the performance scores assuming a level of confidence of 50% in the responses.

The Perimeta model of Strand 2 performance was used to propagate the evidence of Responses

and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender

and by Question).

139

Table 146 (Appendix 5) Strand 2 presents a full list of results covering performance Overall, by

Gender, by Academy and by Question. Results for male and female students are tabulated

separately.

Table 109 presents dashboard summaries of the full results of the Perimeta analysis of the

experience of male and female students.

Table 109 Strand 2 - Results dashboards

Male students Female students

Strand 2 performance analysis by Question

Table 110 illustrates the distribution of all responses for Strand 2 according to the strength of

evidence of self-perception of learning. The horizontal axis represents positive evidence (POS) and

the vertical axis represents negative evidence (NEG), so that the more uncertain (UNC) responses

would appear towards the lower left of the graph. This indicates both the very wide range of

strengths of perceptions (10% to 100%) and in the degree of uncertainty (zero to 60%). The results

show a very strong bias towards positive perceptions, with average positive evidence (73%)

outweighing average uncertainty (20%) and average negative evidence (7%).

Table 110 Strand 2 – Distribution of responses

140

Strand 2 performance analysis by Gender

Table 111 shows a substantial difference overall between the perceptions of male and female

students. For all questions, the combined evidence of positive self-perceptions for males was 86%

and for females only 39%. Both genders show similarly low levels of evidence of negative self-

perceptions of learning. They differ greatly in their uncertainty with males at 13% and females at

55%.

Table 112 compares the ‘top ten’ responses of male and female students, showing in bold font the

common ground between them. Whilst male and female students agreed strongly with many

statements (with almost complete agreement on some), there were only two statements in

common in their top ten positive ratings. Uncertainty in responses was also similar between male

and female students, but again there were few (three) statements in common.

It is in regard to disagreement with statements that there is the greatest difference between male

and female students in terms of scores, and at the same time the greatest similarity in their top ten

statements with six of them in common. It is noted that few statements in either list were

especially strongly disagreed with.

A high degree of correlation between the top ten lists for positive evidence, uncertainty and

negative evidence of self-perceptions of learning. For male students there were seven statements

(six for females) to which responses were both uncertain and negative.

Table 111 Strand 2- Comparison of most substantial responses by male and female students

Top 10 statements by evidence of positive self-perceptions of learning (POS) Male students Female students

Q Statement POS Q Statement POS

S2 9 I am confident in my learning 0.99 S2 4 I like learning new things 1.00

S2 13 I don’t mind making mistakes because I learn from them 0.96

S2 35 We are encouraged to care about the needs of other people in the local community and around the world

1.00

S2 25 My teachers constantly expect me to improve on my personal best 0.96

S2 9 I am confident in my learning 0.99

S2 21 I get on well with my teachers 0.96 S2 2 I feel safe in the Academy 0.99

S2 32 I feel proud about what I have achieved so far 0.96

S2 40 My parents or carers have always been able to sort out any problems that I have had

0.98

S2 16 My teachers teach well so that I learn successfully 0.96

S2 23 I know that the senior teachers want us to learn successfully

0.97

S2 17 I have regular opportunities to express my opinion to my teachers about my learning 0.96

S2 37 My parents or carers feel that they are always welcome in the Academy

0.95

S2 38 My parents or carers feel involved with my learning and my progress 0.96

S2 25 My teachers constantly expect me to improve on my personal best

0.94

S2 18 I often have conversations with my teachers which help me to make better progress 0.96

S2 33 We are taught about the responsibilities of students as well as their rights

0.93

S2 31 I feel that my successes with activities outside the Academy are recognised 0.91

S2 20 My teachers often share things that they have learned themselves

0.91

Table 112 Strand 2 - Top 10 statements by uncertainty in self-perceptions of learning (UNC)

Male students Female students

Q Statement UNC Q Statement UNC

S2 2 I feel safe in the Academy 0.60 S2 31 I feel that my successes with activities outside

the Academy are recognised 0.60

S2 36 I regularly offer to help others 0.37 S2 7 I like working with other students to help my

learning 0.56

141

S2 41 I am encouraged to lead activities in the classroom 0.35

S2 42 I have had opportunities to lead other activities at the Academy

0.55

S2 8 I don’t distract other students 0.34 S2 14 I come up with new ideas to help my learning 0.47

S2 14 I come up with new ideas to help my learning 0.33

S2 41 I am encouraged to lead activities in the classroom

0.36

S2 22 I get extra support to help my learninAcademy 1g when I need it 0.33

S2 19 My teachers seem to enjoy teaching my classes 0.36

S2 30 I feel that my successes at the Academy are recognised 0.33

S2 18 I often have conversations with my teachers which help me to make better progress

0.35

S2 6 I am hard to distract 0.33 S2 8 I don’t distract other students 0.35

S2 40

My parents or carers have always been able to sort out any problems that I have had 0.33

S2 17 I have regular opportunities to express my opinion to my teachers about my learning

0.34

S2 24

Everyone works very hard to make the Academy a great place to learn and to do well 0.33

S2 3 I feel valued and cared for as an individual in the Academy

0.33

Table 113 Strand 2 - Top 10 statements by evidence of negative self-perceptions of learning (NEG)

Male students Female students

Q Statement NEG Q Statement NEG

S2 8 I don’t distract other students 0.28 S2 22 I get extra support to help my learning when

I need it 0.70

S2 14 I come up with new ideas to help my learning 0.17

S2 6 I am hard to distract 0.52

S2 22 I get extra support to help my learning when I need it 0.17

S2 18 I often have conversations with my teachers which help me to make better progress

0.48

S2 30 I feel that my successes at the Academy are recognised 0.16

S2 3 I feel valued and cared for as an individual in the Academy

0.42

S2 6 I am hard to distract 0.15 S2 14 I come up with new ideas to help my

learning 0.26

S2 40

My parents or carers have always been able to sort out any problems that I have had 0.15

S2 17 I have regular opportunities to express my opinion to my teachers about my learning

0.20

S2 24

Everyone works very hard to make the Academy a great place to learn and to do well 0.15

S2 41 I am encouraged to lead activities in the classroom

0.19

S2 1 I feel included in the activities that are available to me in the Academy 0.15

S2 19 My teachers seem to enjoy teaching my classes 0.18

S2 29 I think that my results at the moment are as good as I can do 0.15

S2 8 I don’t distract other students 0.17

S2 41 I am encouraged to lead activities in the classroom 0.10

S2 1 I feel included in the activities that are available to me in the Academy

0.13

Strand 2 performance analysis by Academy

Table 146 (Appendix 2) shows substantial differences between the two Academies taking part in

Strand 2 in terms of the positive outlook of for female students (Academy 1 84% positive, Academy

3 39%) and the uncertainty of females students (Academy 1 14% uncertain, Academy 3 55%). For

male students the difference between Academies is less marked.

Results obtained using the Perimeta model of Strand 2 are compared in Table 114 with the results

of standard statistical tests provided by the ECHO project team.

Table 114 Strand 2 – Comparison of results from Perimeta and statistical tests

Perimeta results Statistical test results Notes

Academy 1 similar to Academy 3 in

evidence of positive self-perception

of learning (male students)

10 of 42 statements with

significantly higher mean responses

for Academy 1, with remaining 32

statements similar in response

Perimeta analysis broadly

consistent with statistical analysis

142

Academy 1 substantially more

positive than Academy 3 in self-

perception of learning (female

students)

22 of 42 statements with

significantly higher mean responses

for female students

Perimeta analysis broadly

consistent with statistical analysis

Evidence of strong bias towards

positive perceptions

Means of scores – none above 3.5

and only one third above 3

Perimeta analysis reveals deeper

insight into positive perceptions

Evidence of negative self-

perception is generally low but with

some anomalies

Means of scores all but one above

2.5

Perimeta analysis broadly

consistent with statistical analysis

and reveals deeper insight into

negative perceptions

Evidence of substantial uncertainty

in many areas of questionnaire

Standard deviation of scores from

questionnaire not excessive

Perimeta uncertainties statistical

standard deviations not related

STRAND THREE - STORIES OF TRANSFORMATION

Strand three was a narrative analysis of a sample of ten students' stories about their experience of

the Academy. It was analysed thematically for evidence of deep, transformative learning and each

case was ranked on a scale of one (low) to four (high).

A full analysis of this small dataset using Perimeta was not meaningful, for the equivalent of one

Question and one Academy and with few students of either Gender. The following Table illustrates

these results in the form of Italian Flag graphics.

Table 115 Strand 3 – Results in Italian flag format

Student Gender Combined Score Italian Flag representation

1 Male 4

2 Female 3

3 Male 3

4 Female 2

5 Female 3

6 Female 2

7 Female 3

8 Male 3

9 Female 2

10 Male 1

11 Female 2

12 Female 2

13 Female 3

143

STRAND FOUR: POST-16 TRANSITION AND PROGRESS TO ADULTHOOD

SUMMARY FROM SOCIAL SCIENCE DATA ANALYSIS

This Strand sought to explore the experiences of leavers from two of the Academies and their

expectations of the transition to the next stage of their education.

When comparing Academies using the traditional social science methods, there were no statistical

differences between the two Academies in 24 out of 30 items indicating the student respondents

had similar views on these statements. However, Academy 1 had lower means in the remaining six

items than Academy 3, where on average, Academy 3 respondents tended to report a greater

degree of agreement on enjoying their time at the Academy, suggesting that Academy 3 students

overall rating their Academy more highly than Academy 1 students, although this was just on 6 of

the 30 items.

There were no statistically significant gender differences in any of the Academies on how students

reported their experiences of the Academy or their expectations of the transition. There were also

no statistically significant differences between students receiving FSM compared to those with no

FSM, suggesting that neither gender nor socioeconomic background was a particular factor

impacting on students’ experiences of their Academies.

PERIMETA ANALYSIS

A total of 4140 data points (for 138 respondents x 30 statements) were entered in the Perimeta models and verified through statistical comparison with the source data.

Performance ratings representing evidence of self-perception of learning were calculated from the

performance scores assuming a level of confidence of 50% in the responses.

The Perimeta model of Strand 4 performance was used to propagate the evidence of Responses

and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender

and by Question).

Table 146 (Appendix 5) presents a full list of results for Strand 2 covering performance Overall, by

Gender, by Academy and by Question.

Table 115 presents dashboard summaries of the full results of the Perimeta analysis of the

experience of students.

144

Table 116 Strand 4 - Results dashboard

Strand 4 performance analysis by Question

Table 116 illustrates the distribution of all responses for Strand 4 according to strength of evidence

of self-perception of learning. The horizontal axis represents positive evidence (POS) and the

vertical axis represents negative evidence (NEG), so that the more uncertain (UNC) responses

would appear towards the lower left of the graph. This indicates both the wide range of strengths

of perceptions (50% to 100%) and in the degree of uncertainty (zero to 40%). The results show a

very strong bias towards positive perceptions, with average positive evidence (84%) outweighing

average uncertainty (14%) and average negative evidence (2%).

Table 117 Strand 4 – Distribution of responses

145

Table 118 Strand 4 - Comparison of most substantial responses

Top 10 statements by evidence of positive self-perceptions of learning (POS)

Q Statement POS

S4 10 I was helped and supported whenever I struggled with my learning 0.98

S4 16 My teachers taught me well at the Academy 0.90

S4 27 The Academy made good use of new technologies for learning 0.90

S4 1 I was valued and included during my time at the Academy 0.89

S4 18 I was regularly challenged to do better so that I could reach my full potential 0.89

S4 24 There were great facilities for learning 0.89

S4 2 I felt part of a community that was focused on learning 0.89

S4 11 I developed a sense of responsibility towards other people during my time at the Academy 0.89

S4 26 I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum 0.89

S4 3 I felt safe at the Academy 0.89

Table 119 Strand 4 - Top 10 statements by uncertainty in self-perceptions of learning (UNC)

Q Statement UNC

S4 13 I supported activities that would improve things for other people 0.38

S4 15 I led activities during my time at the Academy 0.34

S4 7 I am looking forward to the move to Sixth Form, Further Education, training or work 0.33

S4 19 My parents/carers felt involved with my education at the Academy 0.15

S4 22 Everyone at the Academy worked very hard to make it a great place to learn and to do well 0.15

S4 4 I enjoyed my time at the Academy 0.15

S4 30 I could not have gone to a better place for my secondary education 0.14

S4 14 I was offered opportunities for leadership during my time at the Academy 0.14

S4 28 I never felt that anyone was putting limits on what I could achieve 0.13

S4 6 My teachers accurately predicted my GCSE results 0.13

Table 120 Strand 4 - Top 10 statements by evidence of negative self-perceptions of learning (NEG)

Q Statement NEG

S4 7 I am looking forward to the move to Sixth Form, Further Education, training or work 0.18

S4 15 I led activities during my time at the Academy 0.17

No others above 10%

Strand 4 performance analysis by Gender

Table 147 (Appendix 5) shows a substantial difference between genders. Male students were much

more positive (89%) than females (36%) in their self-perceptions of learning. However, females

were not so much more negative but rather more uncertain (56%) than males (11%).

Strand 4 performance analysis by Academy

Table 147 shows that there was no substantial difference between the results obtained from each

of the two participating Academies. Academy 3 students were slightly more positive (94%) than

those of Academy 1 (85%) and slightly less uncertain overall.

Results obtained using the Perimeta model of Strand 4 are compared in Table 120 with the results

of standard statistical tests provided by the ECHO project team.

Table 121 Strand 4 – Comparison of results from Perimeta and statistical tests

Perimeta results Statistical test results Notes

No substantial difference between

Academies in evidence of positive

self-perception of learning

6 of 30 statements with significantly

higher responses for Academy 3,

with remaining 24 statements

Perimeta analysis broadly

consistent with statistical analysis

146

similar in response

Substantial difference between

male and female students in

evidence of positive self-perception

of learning

No significant difference between

responses of male and female

students

Perimeta analysis different from

statistical analysis, and reveals

deeper insight into positive

perceptions

Evidence of strong bias towards

positive perceptions

Means of scores – none above 3.5

but 80% above 3

Perimeta analysis broadly

consistent with statistical analysis

and reveals deeper insight into

positive perceptions

Evidence of negative self-

perception is low

Means of scores above 2.5 for all

statements

Perimeta analysis broadly

consistent with statistical analysis

and reveals deeper insight into

negative perceptions

Evidence of substantial uncertainty

in many areas of questionnaire

Standard deviation of scores from

questionnaire not excessive

Perimeta uncertainties statistical

standard deviations not related

STRAND FIVE: QUALITY OF RELATIONSHIPS BETWEEN PARENTS/CARERS AND THE

ACADEMY

SUMMARY FROM SOCIAL SCIENCE DATA ANALYSIS

This Strand explored how well parents and carers understood and experienced the ECHO charter’s

values in practice.

The findings analysed using traditional social science methods showed that a third of the

statements within the 30 item 2-part questionnaire were generally disagreed with. Many of these

statements seemed to refer to issues of developing attitudes to help others or to take opportunities

to lead activities or referred to behaviour or a healthy lifestyle. However, these findings also

showed these same statements to be among those in which the parents/carers had least

consensus. This suggests that whilst there are general disagreements about certain aspects of how

parents/carers experienced the Oasis Charter, these disagreements do not all concord with one

another and could reflect the diversity of experiences due to the parents’ expectations or

perceptions. These could be affected by other factors such as culture or language, however data on

English as an additional language was missing so such assumptions could be problematic.

PERIMETA ANALYSIS

A total of 1728 data points (72 respondents x 24 statements) were entered in the Perimeta models and verified through statistical comparison with the source data.

147

Performance ratings representing evidence of evidence of self-perception of learning power were

calculated from the performance scores assuming a level of confidence of 50% in the responses.

The Perimeta model of Strand 5 performance was used to propagate the evidence of Responses

and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender

and by Question).

Table 148 (Appendix 5) presents a full list of results for Strand 5 covering performance Overall, by

Academy and by Question. Table 121 presents dashboard summaries of the full results of the

Perimeta analysis of the experience of students.

Table 122 Strand 5 - Results dashboard

Strand 5 performance analysis by Question

Table 122 illustrates the distribution of all responses for Strand 5 according to the strength of

evidence of self-perception of learning. The horizontal axis represents positive evidence (POS) and

the vertical axis represents negative evidence (NEG), so that the more uncertain (UNC) responses

would appear towards the lower left of the graph. This indicates both the very wide range of

strengths of perceptions (20% to 100%) and in the degree of uncertainty (10% to 60%). The results

show a bias towards positive perceptions, with average positive evidence (52%) outweighing

average uncertainty (40%) which is a substantial factor in these responses.

148

Table 123 Strand 5 – Distribution of responses

Table 124 Strand 5 - Comparison of most substantial responses

Strand 5 - Top 10 statements by evidence of positive self-perceptions of learning (POS)

Q Statement BiS

S5 1 My child is known and valued as an individual 0.87

S5 5 My child is taught well 0.59

S5 6 My child is being challenged to improve on previous best performance in all areas 0.59

S5 8 My child’s successes are recognised 0.59

S5 9 The Academy ensures my child is well looked after 0.59

S5 15 I get regular and helpful information about my child’s progress 0.59

S5 20 The Academy is led and managed effectively 0.59

S5 21 My impression is that senior staff focus their work on making the Academy a good place to learn 0.59

S5 22 My impression is that senior staff expect all students to achieve well 0.59

S5 19 The Academy helps me to support my child’s learning 0.59

Table 125 Strand 5 - Top 10 statements by uncertainty in self-perceptions of learning (UNC)

Q Statement UiS

S5 23 My child enjoys their education at the Academy 0.59

S5 2 The Academy is providing for my child’s particular needs 0.57

S5 4 The Academy focuses its work on providing the best opportunities for learning for all students 0.57

S5 10 The Academy helps my child to have a healthy lifestyle 0.57

S5 3 My child feels included in the life of the Academy, in classrooms and in other activities 0.55

S5 14 My child has kept up a good rate of progress at key transitions (for example, from primary to secondary school; from Year 9 to GCSE courses in Years 10 & 11; from Year 11 to Sixth Form/College; from Year 13 to further education/higher education/employment)

0.55

S5 13 My child is taking opportunities to lead activities (for example, leading group work in the classroom, leading a sports team, giving feedback to teachers about their learning, being a representative on a Student Council, taking a lead role in Academy performances of music, drama or dance, being a coach, mentor or ‘buddy’)

0.43

S5 6 My child is being challenged to improve on previous best performance in all areas 0.37

S5 8 My child’s successes are recognised 0.37

149

S5 20 The Academy is led and managed effectively 0.37

Table 126 Strand 5 - Top 10 statements by evidence of negative self-perceptions of learning (NEG)

Q Statement DiS

S5 13 My child is taking opportunities to lead activities (for example, leading group work in the classroom, leading a sports team, giving feedback to teachers about their learning, being a representative on a Student Council, taking a lead role in Academy performances of music, drama or dance, being a coach, mentor or ‘buddy’)

0.33

S5 7 My child’s progress at the Academy has exceeded our expectations 0.27

S5 11 There is a good standard of behaviour at the Academy 0.15

S5 14 My child has kept up a good rate of progress at key transitions (for example, from primary to secondary school; from Year 9 to GCSE courses in Years 10 & 11; from Year 11 to Sixth Form/College; from Year 13 to further education/higher education/employment)

0.10

S5 3 My child feels included in the life of the Academy, in classrooms and in other activities 0.10

S5 24 I would recommend the Academy to another parent/carer 0.08

S5 10 The Academy helps my child to have a healthy lifestyle 0.07

S5 2 The Academy is providing for my child’s particular needs 0.06

S5 4 The Academy focuses its work on providing the best opportunities for learning for all students 0.06

S5 12 My child is developing an attitude of helping others (for example, by doing practical activities to improve life at the Academy, by supporting people in need, getting involved with community projects, fund-raising)

0.06

Results obtained using the Perimeta model of Strand 5 are compared in Table 126 with the results

of standard statistical

Table 127 Strand 5 – Comparison of results from Perimeta and statistical tests

Perimeta results Statistical test results Notes

Evidence of bias towards positive

perceptions

Means of scores – none above 3.5

but majority above 3

Perimeta analysis broadly

consistent with statistical analysis

and reveals deeper insight into

positive perceptions

Evidence of negative self-

perception of child’s learning is low,

but including uncertainty amounts

to approximately half of evidence

Means of scores above 2.5 for all

statements

One third of statements disagreed

with

Perimeta analysis broadly

consistent with statistical analysis

and reveals deeper insight into

negative perceptions

Evidence of substantial uncertainty

in many areas of questionnaire

Standard deviation of scores from

questionnaire not excessive

Perimeta uncertainties statistical

standard deviations not related

STRAND SIX: LEARNING OF TEACHERS: IMPACT OF CPD ON THE QUALITY OF

CLASSROOM PRACTICE

SUMMARY FROM SOCIAL SCIENCE ANALYSIS

This Strand was a three part questionnaire designed to explore teachers’ perceptions of their

Academy and the impact of their professional learning on their classroom practice.

Employing traditional social science methods of data analysis, the first part of the questionnaire

revealed a range of findings with regards to professional learning processes, particularly research

and evaluation and or applying findings of published research to the teachers’ practice. However,

150

teachers agreed on the five most regular processes engaged in by the teacher participants which

referred to improving their teaching and learning practice.

There was, however, a higher degree of variation with regards to the extent with which they felt

involved or engaged in practices which promoted the policy or professional development of the

Academy. But interestingly, teachers tended to show agreement on items which referred to self-

reflection and evaluation or for seeking help for teaching related issues.

The part of the questionnaire which contained items on leadership practices found that the

majority of statements did not frequently apply to practices at the Academy. Much of the findings

from this section seem to suggest that teachers did not feel especially involved in practices which

might promote the staff as part of a whole team, however, staff reported strong commitment to

their departments, key stage or year group.

The findings, nevertheless, did show that teachers felt they were reaching their potential as

teachers and felt presented with opportunities for development with good role models of senior

teachers. What is also interesting is that teachers generally disagreed with the extent to which the

Academy supports parents in the learning of their children and the balancing of

rights/responsibilities within the community. This seemed to concur with the findings of Strand

Five where parents/carers responded in a similar way on items which explored how the Academy

performed as part of a community.

PERIMETA ANALYSIS

A total of 650 data points (for 10 respondents x 65 statements) were entered in the Perimeta models and verified through statistical comparison with the source data.

Performance ratings representing evidence of self-perception of learning were calculated from the

performance scores assuming a level of confidence of 50% in the responses.

The Perimeta model of Strand 6 performance was used to propagate the evidence of Responses

and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender

and by Question).

Table 149 (Appendix 5) presents a full list of results for Strand 6 covering performance Overall, by

Academy and by Question.

Table 127 presents dashboard summaries of the full results of the Perimeta analysis of the

experience of students.

151

Table 128 Strand 6 - Results dashboard

Strand 6 performance analysis by Question

Table 128 illustrates the distribution of all responses for Strand 6 according to the strength of

evidence of self-perception of learning. The horizontal axis represents positive evidence (POS) and

the vertical axis represents negative evidence (NEG), so that the more uncertain (UNC) responses

would appear towards the lower left of the graph. This indicates both the very wide range of

strengths of perceptions (5% to 100%) and in the degree of uncertainty (zero to 60%). The results

show a very strong bias towards positive perceptions, with average positive evidence (66%)

outweighing average uncertainty (23%) and average negative evidence (12%).

152

Table 129 Strand 6 – Distribution of responses

Table 130 Strand 6 - Comparison of most substantial responses

Strand 6 Parts A, B and C - Top 10 statements by positive self-perceptions of learning (POS)

Q Statement POS

S6 A25 My practice is based on the belief that all students are capable of learning 1.00

S6 A10 I experiment with my practice as a conscious strategy for improving classroom teaching and learning 1.00

S6 A1 Staff as well as students learn in this academy 0.92

S6 A11 I modify my practice in the light of feedback from my students 0.89

S6 A14 I modify my practice in the light of evidence from evaluations of my classroom practice by managers or other colleagues

0.87

S6 A27 My experience is that student success is celebrated 0.87

S6 A9 I reflect on my practice as a way of identifying my professional learning needs 0.86

S6 A13 I modify my practice in the light of evidence from self-evaluations of my classroom practice 0.86

S6 A7 I am able to see how practices that work in one context might be adapted to other contexts 0.85

S6 A19 If I have a problem with my teaching I turn to colleagues for help 0.85

Q Statement POS

S6 B1 Senior leaders communicate a clear vision of where the academy is going 0.99

S6 B2 Staff have a commitment to the whole academy as well as to their department, key stage and/or year group 0.92

S6 B16 The academy provides cover to allow staff joint planning time 0.90

S6 B6 Teachers’ professional know-how is used in the formulation of academy policy and goals 0.87

S6 B22 Information is collected from teachers on those aspects of their work that they themselves think they do effectively 0.87

S6 B12 Staff have a good working knowledge of the Academy Development Plan (or your academy’s equivalent document) 0.85

153

S6 B18 Formal training provides opportunities for teachers to develop professionally 0.85

S6 B17 Teachers are encouraged to experiment with new ideas as a way of promoting professional growth 0.84

S6 B8 Opportunities are provided for teachers to critically evaluate academy policy 0.64

S6 B14 Staff development time is used effectively to realise Academy Development Plan priorities 0.62

Q Statement POS

S6 C1 I feel included in a community that values what I can offer to the staff team and to students 0.99

S6 C4 I have had opportunities to work alongside and/or train with colleagues in dother phases to improve students’ progress 0.99

S6 C5 My experience is that senior leaders are role models as ‘leaders of learning’ in the academy 0.98

S6 C2 I believe that I am working in a community that focuses our attention and activities on learning to improve teaching and student outcomes 0.95

S6 C9 There is an emphasis among students and staff on caring about and helping other people in the wider community 0.91

S6 C8 There is an emphasis among students and staff on caring about and helping other people in the academy community 0.90

S6 C12 I believe that I am realising my full potential as a teacher 0.90

S6 C6 Leadership is shared in this academy and opportunities to take on leadership roles are available to all those that want them and have the relevant knowledge and experience 0.81

S6 C7 My experience is that the academy promotes a balance between the rights and the responsibilities of all members of the community 0.72

S6 C3 My experience suggests that other members of staff are involved in training and development activities to improve their classroom practice and student outcomes 0.65

Table 131 Strand 6 Parts A, B and C - Top 10 statements by uncertainty in self-perceptions of learning (UNC)

Q Statement UNC

S6 A20 I suggest ideas or approaches for colleagues to try in class 0.55

S6 A18 I engage in team teaching as a way of improving practice 0.49

S6 A2 I draw on good practice from other schools as a means to further my own professional development 0.43

S6 A28 I discuss with colleagues how students might be helped to learn how to learn 0.38

S6 A5 I consult my students about how they learn most effectively 0.37

S6 A22 I discuss openly with colleagues what and how they are learning 0.36

S6 A17 I observe other colleagues in the classroom and we give each other feedback 0.36

S6 A8 I use insights from my own professional learning to feed into academy policy development 0.33

S6 A12 I modify my practice in the light of published research evidence 0.33

S6 A21 I make collective agreements to test out new ideas 0.33

Q Statement UNC

S6 B9 Staff are involved in evaluating academy policy 0.43

S6 B25 Teacher-initiated networking is an integral element of staff development 0.40

S6 B20 Teachers are helped to develop skills to observe learning as it happens in the classroom 0.37

S6 B3 Senior leaders promote commitment among staff to the whole academy as well as to the department, key stage and/or year group 0.37

S6 B13 Staff see the Academy Development Plan as relevant and useful to learning and teaching 0.35

S6 B19 Teachers are helped to develop skills to assess students’ work in ways that move students on in their learning 0.35

S6 B23 Information is collected from teachers on effective ways they promote learning to learn skills and knowledge among their students 0.35

S6 B10 Staff participate in important decision-making 0.33

S6 B24 Information is collected from teachers on informal teacher networking in which they play an active role 0.33

S6 B8 Opportunities are provided for teachers to critically evaluate academy policy 0.33

Q Statement UNC

S6 C11 The achievements of staff are recognised and celebrated 0.35

S6 C10 The academy is successful in helping parents/carers to support the learning of their children 0.33

S6 C3 My experience suggests that other members of staff are involved in training and development activities to improve their classroom practice and student outcomes 0.25

S6 C7 My experience is that the academy promotes a balance between the rights and the responsibilities of all members of the community 0.21

154

S6 C6 Leadership is shared in this academy and opportunities to take on leadership roles are available to all those that want them and have the relevant knowledge and experience 0.16

No others above 10%

Table 132Strand 6 Parts A, B and C - Top 10 statements by negative self-perceptions of learning (NEG)

Q Statement NEG

S6 A3 I read research reports as one source of useful ideas for improving my practice 0.37

S6 A15 I carry out joint research/evaluation with one or more colleagues as a way of improving my practice 0.36

S6 A2 I draw on good practice from other schools as a means to further my own professional development 0.24

S6 A18 I engage in team teaching as a way of improving practice 0.22

S6 A21 I make collective agreements to test out new ideas 0.16

S6 A23 I use informal opportunities to discuss how children learn 0.16

S6 A12 I modify my practice in the light of published research evidence 0.16

S6 A8 I use insights from my own professional learning to feed into academy policy development 0.14

No others above 10%

Q Statement NEG

S6 B15 Staff development time is used effectively in the academy 0.81

S6 B23 Information is collected from teachers on effective ways they promote learning to learn skills and knowledge among their students 0.51

S6 B24 Information is collected from teachers on informal teacher networking in which they play an active role 0.45

S6 B3 Senior leaders promote commitment among staff to the whole academy as well as to the department, key stage and/or year group 0.44

S6 B25 Teacher-initiated networking is an integral element of staff development 0.38

S6 B10 Staff participate in important decision-making 0.32

S6 B9 Staff are involved in evaluating academy policy 0.25

S6 B20 Teachers are helped to develop skills to observe learning as it happens in the classroom 0.19

S6 B7 Teachers’ professional know-how is used in the formulation of academy policy, even where this leads to a questioning of established rules, procedures and practices 0.16

S6 B21 Senior leaders support teachers in sharing practice with other schools through networking 0.16

Q Statement NEG

S6 C10 The academy is successful in helping parents/carers to support the learning of their children 0.26

S6 C11 The achievements of staff are recognised and celebrated 0.14

No others above 10%

Results obtained using the Perimeta model of Strand 6 are compared in Table 132 with the results

of standard statistical tests provided by the ECHO project team.

Table 133 Strand 6 – Comparison of results from Perimeta and statistical tests

Perimeta results Statistical test results Notes

Evidence of strong bias towards

positive perceptions

Means of scores above 3.5 for

some statements and above 3 for

over 40% of statements

Perimeta analysis broadly

consistent with statistical analysis

and reveals deeper insight into

positive perceptions

Evidence of negative perceptions Means of scores below 2.5 for

nearly half of statements

Perimeta analysis broadly

consistent with statistical analysis

and reveals deeper insight into

negative perceptions

Evidence of substantial uncertainty

in many areas of questionnaire

Standard deviation of scores from

questionnaire not excessive

Perimeta uncertainties statistical

standard deviations not related

155

STRAND SEVEN: IMPACT OF CONTINUED PROFESSIONAL DEVELOPMENT (CPD) ON THE

QUALITY OF LEADERSHIP AND MANAGEMENT AND CLASSROOM PRACTICE

This Strand explored senior leaders’ perceptions from all three Academies and how they are viewed

to be fulfilling the OCL Charter.

The findings from traditional social science analysis methods of the 55 item questionnaire seemed

to suggest that senior management teams (SMT) perceive an unfriendly macro working

environment in terms of relevant governmental and Academy based policy measures in supporting

teaching and learning that focuses on student achievement. However, SMT seemed to

demonstrate a confidence in taking up the responsibility for their professional development.

The findings also implied the issues of fairness and democracy across Academies in relation to

opportunity being created for improvement of teaching and learning through self-reflection,

participation in relevant professional development activities, use and analysis of data, engagement

in external partnerships and involvement of students in decision making.

Principal Components Analysis (PCA) was performed to explore possible underlying patterns in SMT

participants’ responses to the Likert-scale items by reducing the number of items and

dimensionality, where four underlying themes were revealed.

PERIMETA ANALYSIS

A total of 1350 data points (for 25 respondents x 54 statements) were entered in the Perimeta models and verified through statistical comparison with the source data.

Performance ratings representing evidence of self-perception of learning were calculated from the

performance scores assuming a level of confidence of 50% in the responses.

The Perimeta model of Strand 7 performance was used to propagate the evidence of Responses

and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender

and by Question).

Table 150 presents a full list of results for Strand 7 covering performance Overall, by Academy and

by Question.

Table 133 presents dashboard summaries of the full results of the Perimeta analysis of the

experience of students.

156

Table 134 Strand 7 - Results dashboard

Strand 7 performance analysis by Question

Table 134 illustrates the distribution of all responses for Strand 7 according to evidence of self-

perception of learning. The horizontal axis represents positive evidence (POS) and the vertical axis

represents negative evidence (NEG), so that the more uncertain (UNC) responses would appear

towards the lower left of the graph. This indicates both the wide range of strengths of perceptions

(25% to 100%) and in the degree of uncertainty (zero to 60%). The results show a very strong bias

towards positive perceptions, with average positive evidence (75%) outweighing average

uncertainty (23%) and average negative evidence (3%).

Table 135 Strand 7 – Distribution of responses

157

Table 136 Strand 7 - Comparison of most substantial responses

Strand 7 - Top 10 statements by positive self-perceptions of learning (POS)

Q Statement POS

S7 1 I have a good understanding of my own and other people’s learning 1.00

S7 5 I understand the differences between good and outstanding lessons 1.00

S7 2 I keep my knowledge about teaching and learning and about my phase and/or subject specialism up to date 1.00

S7 51 Good personal relationships and strong teams help us to face challenges with confidence 0.98

S7 15 The senior leadership team communicates a clear vision of where the academy is going so that there is a shared sense of purpose

0.90

S7 17 The senior leadership team effectively promotes commitment among staff to the future development of the whole academy

0.90

S7 24 There are processes for involving students in decision-making, including how best to improve classroom practice 0.90

S7 42 We have efficient and effective support systems and structures across the academy 0.90

S7 52 The senior leadership team has the skills and resilience to successfully work through the more complex challenges that we are facing

0.90

S7 53 The senior leadership team is good at long term planning for capacity building (for example, successfully addressing issues like student progress across key transitions; improving the ‘tail’ of student underachievement and/or significant achievement gaps; succession planning)

0.90

Table 137 Strand 7 - Top 10 statements by uncertainty in self-perceptions of learning (UNC)

Q Statement UNC

S7 8 I regularly participate in critical and reflective enquiry to develop teaching and learning across the academy 0.57

S7 12 I regularly interact with staff in other schools, academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our academy

0.48

S7 18 There is effective communication between senior leaders and teachers 0.38

S7 19 There are processes for involving all staff in decision-making 0.38

S7 25 Staff have a good working knowledge of the Academy Development Plan 0.38

S7 47 The academy involves a wide range of other stakeholders directly and powerfully in the education of students 0.38

S7 20 Teachers’ professional know-how is used in the formulation of academy policy and goals 0.38

S7 23 Staff participate in important decision-making 0.38

S7 26 Staff see the Academy Development Plan as relevant and useful to learning and teaching 0.38

S7 28 Staff development time is used effectively in the academy, particularly through the explicit modelling of good practice

0.36

Table 138 Strand 7 - Top 10 statements by negative self-perceptions of learning (NEG)

Q Statement DIS

S7 12 I regularly interact with staff in other schools, academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our academy

0.24

S7 39 ‘Learning how to learn’ is an issue discussed in staff development time 0.13

No others above 10%

Strand 7 performance analysis by Academy

158

Table 150 (Appendix 5) shows only minor differences between the responses of leaders of the three

Academies.

Results obtained using the Perimeta model of Strand 7 are compared in Error! Reference source

not found. 138 with the results of standard statistical tests provided by the ECHO project team.

Table 139 Strand 7 – Comparison of results from Perimeta and statistical tests

Perimeta results Statistical test results Notes

Evidence of self-perception of

learning

Means of scores from questionnaire Perimeta analysis broadly

consistent with statistical analysis

Evidence of strong bias towards

positive perceptions

Means of scores above 3.5 for

relatively high proportion of

statements

Perimeta analysis broadly

consistent with statistical analysis

and reveals deeper insight into

positive perceptions

Some evidence of negative

perceptions

Means of scores all above 2.5

suggesting no disagreement with

statements

Perimeta analysis reveals deeper

insight into negative perceptions

Evidence of substantial uncertainty

in many areas of questionnaire

Standard deviation of scores from

questionnaire not excessive

Perimeta uncertainties statistical

standard deviations not related

STRAND EIGHT AND NINE: CONTEXT FOR STUDENT LEARNING - KEY PERFORMANCE

INDICATORS AND GCSE EXAM RESULTS

Perimeta analysis was not appropriate for Strands 8 and 9 because the data relating to schools and

GCSE results are purely quantitative, with no qualitative element.

Step Four of the ECHO project will explore how Academies can use KPI and GCSE results to deepen

the insights from Perimeta analyses of student self-perceptions of learning in Strands 1, 2, 3 and 4.

TRIANGULATION: HPM AND SOCIAL SCIENCE

The same data were used for analysis by traditional social science methods and for Perimeta

modelling. After both analyses had been completed a process of triangulation was undertaken in

order to check the validity of the findings across the different types of analysis. To do this the

highest scored items from the Perimeta analysis and the descriptive statistics were compared.

Because Perimeta accounted for certainty and uncertainty while the descriptive statistics ranked

the absolute scores, an exact mapping was not excepted. Rather a broad correlation was expected

because the same evidence was used in a radically different measurement model.

For example, Strand 5, item 15 N=135 the Mean = 2.81 Standard Deviation = 0.75, while in the

Perimeta model Positive Certainty =0.272 Uncertainty = 0.348 and Negative Certainty = 0.38

159

Perimeta makes assumptions about uncertainty based on a performance judgment and a

confidence judgment. One assumption we made was that a large degree of uncertainty would

correlate with a large standard deviation. A second assumption was that a ranking of the most

agreed with or disagreed with items would be broadly similar.

Examples from each of the outcomes from each of the Strands was compared on these

assumptions. There was an expected degree of correlation between what was certain/high in all the

data, but less correlation between what was uncertain and the measures of standard deviation.

WHAT CAN WE LEARN FROM THESE TWO APPROACHES

The Perimeta model shows that 'there are a number of people in the middle that we need to get to

know better' and thus it flags up areas for development and further exploration. The Social Science

model presents a more 'black and white' analysis - 39% disagree and 69% agree, whereas the

Perimeta model shows that there are more people we should be concerned about than the 39%.

The real benefits of Perimeta come in the next stage of the process because it is designed to draw

people into exploring and self-managing their own data. It is essentially a decision making tool, not

only a measurement tool. It measures for the purpose of stimulating positive change, according to a

locally defined purpose.

DISCUSSION OF FINDINGS

ECHO STRAND 1 - DEVELOPMENT OF STUDENTS AS LEARNERS IN YEAR 7

The Integrated Approach applied to Strand 1 has revealed key insights into the strength of self-

perception of the learning power of students in Year 7. There were clear differences between the

performances of Academies in terms of the changes their students perceived over the course of the

year. Differences in perceptions between male and female students were insubstantial. The

Resilience dimension of student learning power decreased whilst increases were seen in the other

six ELLI dimensions.

Resilience was consistently the lowest rated dimension, or in the lowest rated group, for male and

female students and Time 1 and Time 2. Resilience is known through experience in the use of the

ELLI framework to be a complex measure of learning power, not directly related to other ELLI

measures. It would not be appropriate for students to be resilient in their attitude to negative

behaviours, hence low or worsening Resilience in Year 7 is not necessarily a bad sign. This aspect of

the design of the Strand 1 research will be reassessed in Phase 2 of the ECHO project.

160

Results representing strength of self-perception of learning power are broadly consistent between

the Integrated Approach and statistical methods. This comparison has helped to build confidence in

both sets of analyses. However, validation of results in the conventional sense proved to be

challenging because the Integrated Approach is not a substitute for more familiar statistical

techniques but rather a complement to them. Detailed reviews with key stakeholders including

members of the ECHO project team and senior Oasis Academy leaders were necessary to build

confidence in the approach and its results, focusing on:

Understanding the theoretical difference between uncertainty and statistical variation;

Recognising the existence of uncertainty in the evidence provided by student responses;

and

Appreciating that insights into uncertainty as well as positive and negative evidence of self-

perception of learning power could enhance the development of improvements in student

experiences in Year 7.

Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative

responses to the Strand 1 questionnaire. Even assuming a relative high level of confidence (75%) in

the evidence, uncertainty was demonstrated to be a substantial factor accounting for around one

third of the evidence obtained, and was not greatly changed over the course of the year.

The Integrated Approach revealed a different perspective from the statistical analysis of the

performance of Academies, showing that they converged in performance terms from substantially

different starting points (whilst the statistical analysis suggested the Academies had diverged from

similar starting points). The finding, backed up by the statistical analysis, that Academy 1 improved

the most of the three, is consistent with Academy 1’s history of involvement with learning

frameworks similar to ELLI. This hypothesis will be tested in Phase 2 of the ECHO project.

Key insights established in Phase 1 of the research (and described above) would provide the basis

for deeper engagement and solution development with students in Phase 2, exploring the reasons

behind responses to the ELLI questionnaire that are exceptionally positive, or negative, or

uncertain. Error! Reference source not found. identifies a number of areas for further investigation

in Phase 2.

In Phase 1 the Integrated Approach has established for Strand 1 the value of additional insights,

complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2

the Perimeta model for Strand 1 will be further enhanced by ‘drilling down’ to the underlying

causes of positive and negative perceptions. This enhancement will be guided by the priority areas

identified in Table 139 and will help to analyse the more detailed underlying data that will be

obtained from the ELLI learning framework.

Table 140 Strand 1 – Areas for further investigation in Phase 2

Q Statement Time Gender Academy

161

Evidence of positive self-perception of learning > 75%

S1 1 Changing & Learning T1, T2 G2 A1, A2, A3

S1 6 Meaning Making T2 G2 A1, A2, A3

S1 5 Creativity T2 G2 A1, A2, A3

S1 7 Critical Curiosity T2 G1, G2 A1, A2, A3

Evidence of negative self-perception of learning > 25%

S1 4 Resilience T1, T2 G1, G2 A1, A2, A3

S1 5 Creativity T1 G1 A1, A2, A3

S1 3 Strategic Awareness T1 G1 A1, A2, A3

Uncertainty in self-perception of learning > 25%

All T1, T2 G1, G2 A1, A2, A3

High degree change from Time 1 to Time 2

S1 3 Strategic Awareness G1 A1, A2, A3

S1 5 Creativity G1, G2 A1, A2, A3

S1 6 Meaning Making G2 A1, A2, A3

S1 7 Critical Curiosity G2 A1, A2, A3

ECHO STRAND 2 - STUDENT PERCEPTIONS OF THE IMPACT OF THE EDUCATION CHARTER

The Integrated Approach applied to Strand 2 has revealed key insights into the self-perception of

students in Years 8 and 10 of their development as learners. There were clear differences between

the two participating Academies and between the genders in the strength of self-perceptions of

learning. The particular statements that students most readily agreed with, or were uncertain

about, were substantially different between males and females.

Results representing strength of self-perception of learning power are broadly consistent between

the Integrated Approach and statistical methods. This comparison has helped to build confidence in

both sets of analyses. However, validation of results in the conventional sense proved to be

challenging because the Integrated Approach is not a substitute for more familiar statistical

techniques but rather a complement to them. Detailed reviews with key stakeholders including

members of the ECHO project team and senior Oasis Academy leaders were necessary to build

confidence in the approach and its results, focusing on:

Understanding the theoretical difference between uncertainty and statistical variation;

Recognising the existence of uncertainty in the evidence provided by student responses;

and

Appreciating that insights into uncertainty as well as positive and negative evidence of self-

perception of learning could enhance the development of improvements in student

experiences in Year 7.

Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative

responses to the Strand 2 questionnaire. Uncertainty was demonstrated to be a substantial factor

accounting for around one fifth of the evidence obtained.

Key insights established in Phase 1 of the research (and described above) would provide the basis

for deeper engagement and solution development with students in Phase 2, exploring the reasons

162

behind responses to the ELLI questionnaire that are exceptionally positive, or negative, or

uncertain. Table 89 identifies a number of areas for further investigation in Phase 2.

In Phase 1 the Integrated Approach has established for Strand 2 the value of additional insights,

complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2

the Perimeta model for Strand 2 will be further enhanced by ‘drilling down’ to the underlying

causes of positive and negative perceptions. This enhancement will be guided by the priority areas

and themes identified in Table 140, and by themes revealed by statistical factor analysis, and will

help to analyse the more detailed underlying data.

Table 141 Strand 2 – Statements for further investigation in Phase 2

Q Statement Gender Academy

Evidence of positive self-perception of learning > 75%

S2 4 I like learning new things G2 A1, A2, A3

S2 35 We are encouraged to care about the needs of other people in the local community and around the world

G2 A1, A2, A3

S2 9 I am confident in my learning G2 A1, A2, A3

S2 2 I feel safe in the Academy G2 A1, A2, A3

S2 9 I am confident in my learning G1 A1, A2, A3

Uncertainty in self-perception of learning > 50%

S2 2 I feel safe in the Academy G1 A1, A2, A3

S2 31 I feel that my successes with activities outside the Academy are recognised G2 A1, A2, A3

S2 7 I like working with other students to help my learning G2 A1, A2, A3

S2 42 I have had opportunities to lead other activities at the Academy G2 A1, A2, A3

Evidence of negative self-perception of learning > 25%

S2 22 I get extra support to help my learning when I need it G2 A1, A2, A3

S2 6 I am hard to distract G2 A1, A2, A3

S2 18 I often have conversations with my teachers which help me to make better progress G2 A1, A2, A3

S2 3 I feel valued and cared for as an individual in the Academy G2 A1, A2, A3

S2 8 I don’t distract other students G1 A1, A2, A3

S2 14 I come up with new ideas to help my learning G2 A1, A2, A3

Strong/ common themes in responses

Student: confidence, learning from mistakes, learning new things, pride, responsibility

Teacher: quality of teaching, expectations, sharing learning

School/ leader: feeling safe

Community/ parent: recognition of activities outside

ECHO STRAND 3 – STORIES OF TRANSFORMATION

A full analysis of this small dataset using Perimeta was not meaningful, for the equivalent of one

Question and one Academy and with few students of either Gender. In Step Four of the ECHO

project, when the pilot study will be widened to many more Academies, the Perimeta analysis is

expected to be highly appropriate as an enabler of insights into the reasons behind the diversity of

students’ stories.

ECHO STRAND 4 - POST-16 TRANSITION AND PROGRESS TO ADULTHOOD

The Integrated Approach applied to Strand 4 has revealed key insights into the self-perception of

learning of students in the post-16 transition and progression to adulthood. There were clear

163

differences between male and female students in the strength of self-perceptions of learning. The

particular statements that students most readily agreed with, or were uncertain about, were

substantially different between males and females.

Results representing strength of self-perception of learning power are broadly consistent between

the Integrated Approach and statistical methods in comparing Academies, but different in

comparing genders. Validation of results in the conventional sense proved to be challenging

because the Integrated Approach is not a substitute for more familiar statistical techniques but

rather a complement to them. Detailed reviews with key stakeholders including members of the

ECHO project team and senior Oasis Academy leaders were necessary to build confidence in the

approach and its results, focusing on:

Understanding the theoretical difference between uncertainty and statistical variation;

Recognising the existence of uncertainty in the evidence provided by student responses;

and

Appreciating that insights into uncertainty as well as positive and negative evidence of self-

perception of learning could enhance the development of improvements in student

experiences in Year 11.

Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative

responses to the Strand 4 questionnaire. Uncertainty was demonstrated to be a substantial factor

accounting for around one seventh of the evidence obtained.

Key insights established in Phase 1 of the research (and described above) would provide the basis

for deeper engagement and solution development with students in Phase 2, exploring the reasons

behind responses to the questionnaire that are exceptionally positive, or negative, or uncertain.

Table 90 identifies a number of areas for further investigation in Phase 2.

In Phase 1 the Integrated Approach has established for Strand 4 the value of additional insights,

complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2

the Perimeta model for Strand 4 will be further enhanced by ‘drilling down’ to the underlying

causes of positive and negative perceptions. This enhancement will be guided by the priority areas

and themes identified in Table 141, and by themes revealed by statistical factor analysis, and will

help to analyse the more detailed underlying data.

Table 142 Strand 4 – Statements for further investigation in Phase 2

Q Statement Gender Academy

Evidence of positive self-perception > 98%

S4 10 I was helped and supported whenever I struggled with my learning G1 & G2 A1, A2, A3

Uncertainty in self-perception of learning > 50%

None None

Evidence of negative self-perception > 25%

none None

Strong/ common themes in responses

Student:

164

Teacher: support, quality of teaching, challenging

School/ leader: valued and included, community, technology, facilities

Community/ parent:

ECHO STRAND 5 – QUALITY OF RELATIONSHIPS BETWEEN PARENTS/CARERS AND THE

ACADEMY

The Integrated Approach applied to Strand 5 has revealed key insights into the self-perception of

parents/ carers of the learning of their children.

Results representing strength of self-perception of learning are broadly consistent between the

Integrated Approach and statistical methods. Validation of results in the conventional sense proved

to be challenging because the Integrated Approach is not a substitute for more familiar statistical

techniques but rather a complement to them. Detailed reviews with key stakeholders including

members of the ECHO project team and senior Oasis Academy leaders were necessary to build

confidence in the approach and its results, focusing on:

Understanding the theoretical difference between uncertainty and statistical variation;

Recognising the existence of uncertainty in the evidence provided by parent/carer

responses; and

Appreciating that insights into uncertainty as well as positive and negative evidence of self-

perception of learning power could enhance the development of improvements in

parent/carer experiences of their children’s learning.

Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative

responses to the Strand 5 questionnaire. Uncertainty was demonstrated to be a substantial factor

accounting for around 40 percent of the evidence obtained.

Key insights established in Phase 1 of the research (and described above) would provide the basis

for deeper engagement and solution development with parents/ carers in Phase 2, exploring the

reasons behind responses to the questionnaire that are exceptionally positive, or negative, or

uncertain. Error! Reference source not found.le 91 identifies a number of areas for further

investigation in Phase 2.

In Phase 1 the Integrated Approach has established for Strand 4 the value of additional insights,

complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2

the Perimeta model for Strand 5 will be further enhanced by ‘drilling down’ to the underlying

causes of positive and negative perceptions. This enhancement will be guided by the priority areas

and themes identified in Table 142 and by themes revealed by statistical factor analysis, and will

help to analyse the more detailed underlying data.

165

Table 143 Strand 5 – Statements for further investigation in Phase 2

Q Statement Gender Academy

Evidence of positive self-perception of learning > 98%

None

Uncertainty in self-perception of learning >50%

S5 23 My child enjoys their education at the Academy

S5 2 The Academy is providing for my child’s particular needs

S5 4 The Academy focuses its work on providing the best opportunities for learning for all students

S5 10 The Academy helps my child to have a healthy lifestyle

S5 3 My child feels included in the life of the Academy, in classrooms and in other activities

S5 14 My child has kept up a good rate of progress at key transitions (for example, from primary to

secondary school; from Year 9 to GCSE courses in Years 10 & 11; from Year 11 to Sixth

Form/College; from Year 13 to further education/higher education/employment)

Evidence of negative self-perception of learning > 25%

S5 13 My child is taking opportunities to lead activities (for example, leading group work in the

classroom, leading a sports team, giving feedback to teachers about their learning, being a

representative on a Student Council, taking a lead role in Academy performances of music,

drama or dance, being a coach, mentor or ‘buddy’)

S5 7 My child’s progress at the Academy has exceeded our expectations

Strong/ common themes in responses

Student:

Teacher: quality of teaching, challenging

School/ leader: valued, recognised, looked after, informed, effective leadership

Community/ parent:

ECHO STRAND 6 - LEARNING OF TEACHERS: IMPACT OF CPD ON THE QUALITY OF

CLASSROOM PRACTICE

The Integrated Approach applied to Strand 6 has revealed key insights into the self-perception of

learning of teachers.

Validation of results in the conventional sense proved to be challenging because the Integrated

Approach is not a substitute for more familiar statistical techniques but rather a complement to

them. Detailed reviews with key stakeholders including members of the ECHO project team and

senior Oasis Academy leaders were necessary to build confidence in the approach and its results,

focusing on:

Understanding the theoretical difference between uncertainty and statistical variation;

Recognising the existence of uncertainty in the evidence provided by student responses;

and

Appreciating that insights into uncertainty as well as positive and negative evidence of self-

perception of learning could enhance the development of improvements in teacher

experiences.

166

Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative

responses to the Strand 6 questionnaire. Uncertainty was demonstrated to be a substantial factor

accounting for around one quarter of the evidence obtained.

Key insights established in Phase 1 of the research (and described above) would provide the basis

for deeper engagement and solution development with students in Phase 2, exploring the reasons

behind responses to the questionnaire that are exceptionally positive, or negative, or uncertain.

Table 92 identifies a number of areas for further investigation in Phase 2.

In Phase 1 the Integrated Approach has established for Strand 6 the value of additional insights,

complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2

the Perimeta model for Strand 6 will be further enhanced by ‘drilling down’ to the underlying

causes of positive and negative perceptions. This enhancement will be guided by the priority areas

and themes identified in Table 143, and by themes revealed by statistical factor analysis, and will

help to analyse the more detailed underlying data.

Table 144 Strand 6 – Statements for further investigation in Phase 2

Q Statement

Evidence of positive self-perception of learning >98%

S6 A25 My practice is based on the belief that all students are capable of learning

S6 A10 I experiment with my practice as a conscious strategy for improving classroom teaching and learning

S6 B1 Senior leaders communicate a clear vision of where the academy is going

S6 C1 I feel included in a community that values what I can offer to the staff team and to students

S6 C4 I have had opportunities to work alongside and/or train with colleagues in dother phases to improve students’ progress

S6 C5 My experience is that senior leaders are role models as ‘leaders of learning’ in the academy

Uncertainty in self-perception of learning > 25%

S6 A20 I suggest ideas or approaches for colleagues to try in class

Evidence of negative self-perception of learning >25%

S6 B15 Staff development time is used effectively in the academy

S6 B23 Information is collected from teachers on effective ways they promote learning to learn skills and knowledge among their students

Strong/ common themes in responses

Student: teacher learning, reflection, collaboration, belief in student capability,

Teacher: innovation,

School/ leader: celebrating success, support, communication, commitment, engagement, training

Community/ parent:

ECHO STRAND 7 - IMPACT OF CONTINUED PROFESSIONAL DEVELOPMENT (CPD) ON THE

QUALITY OF LEADERSHIP AND MANAGEMENT AND CLASSROOM PRACTICE

The Integrated Approach applied to Strand 7 has revealed key insights into the self-perception of

learning of leaders.

Validation of results in the conventional sense proved to be challenging because the Integrated

Approach is not a substitute for more familiar statistical techniques but rather a complement to

them. Detailed reviews with key stakeholders including members of the ECHO project team and

167

senior Oasis Academy leaders were necessary to build confidence in the approach and its results,

focusing on:

Understanding the theoretical difference between uncertainty and statistical variation;

Recognising the existence of uncertainty in the evidence provided by student responses;

and

Appreciating that insights into uncertainty as well as positive and negative evidence of self-

perception of learning could enhance the development of improvements in teacher

experiences.

Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative

responses to the Strand 7 questionnaire. Uncertainty was demonstrated to be a substantial factor

accounting for around one quarter of the evidence obtained.

Key insights established in Phase 1 of the research (and described above) would provide the basis

for deeper engagement and solution development with students in Phase 2, exploring the reasons

behind responses to the questionnaire that are exceptionally positive, or negative, or uncertain.

Table 93 identifies a number of areas for further investigation in Phase 2.

In Phase 1 the Integrated Approach has established for Strand 7 the value of additional insights,

complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2

the Perimeta model for Strand 7 will be further enhanced by ‘drilling down’ to the underlying

causes of positive and negative perceptions. This enhancement will be guided by the priority areas

and themes identified in Table 144, and by themes revealed by statistical factor analysis, and will

help to analyse the more detailed underlying data.

Table 145 Strand 7 – Statements for further investigation in Phase 2

Q Statement

Evidence of positive self-perception of learning >98%

S7 1 I have a good understanding of my own and other people’s learning

S7 5 I understand the differences between good and outstanding lessons

S7 2 I keep my knowledge about teaching and learning and about my phase and/or subject specialism up to date

S7 51 Good personal relationships and strong teams help us to face challenges with confidence

Uncertainty in self-perception of learning >50%

S7 8 I regularly participate in critical and reflective enquiry to develop teaching and learning across the academy

Evidence of negative self-perception of learning >25%

None

Strong/ common themes in responses

Student: engagement with students

Teacher: engagement with teachers

School/ leader: insight into people and lesson quality, good relationships, dealing with complex challenges, long-term planning

Community/ parents:

PRACTICAL AND TECHNICAL ISSUES FOR DEVELOPMENT

168

The project proved to be more demanding in terms of resource than was anticipated for three

reasons. First it was essentially a proof of concept. The task was to explore whether hierarchical

process modelling and the Perimeta tool in particular would be useful in supporting the evaluation

of the wider outcomes of schooling. For this reason, and secondly, we chose to compare traditional

methods and Perimeta modelling in analysis of the data, and this was productive but time

consuming. Thirdly, the bespoke surveys were designed to fit the HPM model, and data for these

surveys was collected by each school on line on their own intranet service, and then compiled by

the research team into the core data sets. These data were then entered manually into the

Perimeta model. There was a significant attrition rate on data collection because of this.

Any development of this approach to school self evaluation would require significantly more

automation than was possible in this pilot, as well as agreement around a systems design which is

shared with a group of schools, with the possibility of the addition of a bespoke line of enquiry to

suit local school needs. Therefore further applications of this model would not require the

selection and construction of surveys which was necessary in the pilot.

This pilot study has produced surveys which can be significantly and productively reduced in size,

with new variables being computed from scales which have already been validated. This should

reduce the amount of data to be entered into the Perimeta model whilst also ensuring that it has

more power as a measurement tool, rated with a higher confidence score in the translation into the

Italian Flag model.

With advances in technology and learning analytics it is within our power now to create an online

learning environment which both provides survey tools, and knowledge structuring tools, such as

Perimeta. Outputs of surveys could be undertaken by the school community on a planned basis

throughout the year and be automatically entered into the Perimeta model. This then becomes a

leadership decisioning tool available to leaders throughout the academic year, and particularly at

key stages of development planning cycle.

However, this proof of concept has simply indicated that the research question which we set out

with is valuable and worth pursuing. Beyond what is reported in this report is a second phase

where the data and findings are made available to the schools concerned and the Multi-Academy

Trust to explore their value in networked improvement community processes. If, for example,

there is a key issue which emerges about the need for a holistic and system wide approach to

curriculum and pedagogy in order to achieve the purpose of all students taking responsibility for

their own learning, then this complex issue can become a focus for school improvement. Consistent

with the philosophy underlying Perimeta, of responsible self-management, diversity, participation

and responding to emergence, these issues could become a focus for authentic professional

enquiry. A promising model to use for this, which stimulates professional learning and research, is

Design Educational Engineering and Development which has been developed by the Carnegie

Foundation for the Advancement of Teaching (Bryk et al., 2010, Bryk, 2009, Bryk and Gomez, 2008).

In this model a school or community of schools identifies a shared, complex problem and proceeds

169

through a process of rapid prototyping research and evaluation, led by teams of teachers in 24

week cycles. Whilst teachers undertake their own evaluation and analysis of their interventions

which are designed to address the chosen complex issue, the whole system commits to a shared

disciplined evaluation framework. Meanwhile the collective intelligence which emerges from these

prototypes is harnessed by school leadership - and academic critical friends - and shared across the

network and beyond. Clearly this model is consistent with the purpose of Perimeta. A Perimeta

model may also provide a really useful disciplined evaluation framework which feeds into the

overall school self evaluation and performance management.

These ideals are a long way beyond the completion of this proof of concept pilot study. However,

conceptually, practically, professionally and technically there is real promise in this approach.

CONCLUSIONS AND NEXT STEPS

The introduction to this report describes the challenge taken up by the ECHO Project and the

participating academies in trying to find systematic, reliable and sustainable ways of evaluating the

wider outcomes of schools. Although the separate components of the Project, for example student

and parent questionnaires, are not new tools for school self-evaluation, the pilot phase of the ECHO

Project has been an attempt to weave all the evaluation Strands together in order to create a whole

and more balanced snapshot of the school’s performance for improvement planning.

The pilot phase of the ECHO Project between 2010 and 2013 was designed as a ‘proof of concept’

exercise. It is encouraging to note from the previous section of the report that initial analyses

suggest that ‘conceptually, professionally, practically and technically there is real promise in this

approach’. The use of the Perimeta model has been an important feature of the Project as it

enables links to be established between extended self-evaluation, performance management and

school improvement planning.

The need for further research has been highlighted throughout the report and, in particular, the

relationship between quantitative and qualitative perspectives on performance would benefit from

deeper investigation. For example, what might be the relationships between Year 11 leaver

responses to the Strand 4 questionnaire and GCSE results (Strand 9)? Is there a link between the

responses of students to the Strand 1, Strand 2 and Strand 3 questions and attendance figures for

the same students (Strand 8)?

The pilot phase has helped to test, through traditional statistical techniques, the quality of

questions used in the various questionnaires and interview schedules. As a result, it is suggested

that these can be refined and the amount of data that they generate reduced. It has also prompted

the need to develop further, and possibly build a 360-degree perspective into, the evaluation of

teacher learning and leadership learning. These are both relatively new areas for the evaluation of

the impact of learning on the effectiveness of teachers and leaders.

170

The more systematic and broadly based evaluation developed through the ECHO Project has the

potential to enhance the depth and range of information available to senior leadership teams as

they undertake school improvement planning. The ECHO evaluation processes can be tailored to

the context, purposes and needs of individual schools. The results could also generate issues arising

from and shared by a group of schools rather than a single school. This would benefit from a

coordinated approach to improvement by, for example, a multi-academy group or a local authority

and create opportunities to share knowledge across a group of schools through a networked

improvement community.

A unifying idea behind the ECHO Project has been the quality of the relationships between the

learning of leaders, the learning of teachers and the learning of students as, working together, they

seek to secure outstanding outcomes for students and continuous improvement of their schools.

Insights from parents/carers about the learning of their children and the effectiveness of the school

were added to the data that was analysed by statistical methods and the use of Perimeta. Oasis is

currently developing two additional questionnaires for governors and the wider community in

order to extend the base of evaluation evidence and involve all key stakeholders.

Perimeta is a novel data analysis programme in that it enables leaders to identify areas of

uncertainty during the process of self-evaluation as well as aspects of performance that are more

black and white, or red and green to use the Italian flag metaphor. Clearly weaknesses (the red

indicators) need to be addressed but schools will also benefit from exploring areas of uncertainty in

more depth. To quote from a section earlier in the report Perimeta is designed to draw people into

exploring and managing their own data. It is essentially a decision-making tool, not only a

measurement tool. It measures for the purpose of stimulating positive change according to locally

defined need’.

The results of the pilot phase of the ECHO Project have led the core team to the conclusion that

both the concept and the approach are worth pursuing further. The following are

recommendations for the next steps:

1. The results will be shared with the leadership teams and staff of the three academies that

participated in the pilot project; their feedback will help to frame the next phase of research

and development.

2. The questionnaires and interview schedules used in the pilot phase will be refined, put

online and gradually rolled out across the Oasis group of academies; an additional

questionnaire for governors is being piloted during 2013/14 and a questionnaire for the

wider community developed for initial use in 2014.

3. Senior leadership teams in schools and academies, multi-academy groups, local authorities

and all those involved with raising standards in schools will be encouraged to explore the

171

potential of this more systematic and rigorous approach to evaluating the wider outcomes

of schools for strategic planning and improving student outcomes.

4. Further development will be undertaken on the Perimeta model with the intention of

making it directly available to schools.

5. Further research will be commissioned on a number of issues arising from the pilot phase of

the ECHO Project.

6. The ECHO Project is already creating international interest and there will be wider

dissemination of the results.

7. It is expected that additional suggestions for Phase 2 of the ECHO Project will arise as the

results of the pilot phase are shared more widely.

172

BIBLIOGRAPHY AND REFERENCES

Amosa, W., Ladwig, J., Griffiths, T. & Gore, J. 2007. Equity effects of Quality Teaching: Closing the gap. Australian Association for Research in Education conference. Fremantle.

Anderson, C. S. 1982. The Search for School Climate: A Review of the Research. Review of Educational Research, 52, 368-420.

Argyris, C., & Schön, D. A. (1996). Organizational learning. 2. Theory, method, and practice. Addison-Wesley

Assessment Reform Group 1999. Assessment for Learning: Beyond the Black Box, Cambridge, University of Cambridge School of Education.

Bateson, G. 1972. Steps to an Ecology of Mind, San Fancisco,, Chandler.

Baxter Magolda, M. B. 2004. Evolution of a constructivist conceptualization of epistemological reflection. Educational Psychologist, 39, 31-42.

Bazeley, P. (2007). Qualitative analysis with NVivo. London: Sage.

Beare, H. 2001. Creating the Futures School, London, Routledge.

Bereiter, C. & Scardamalia, M. 1989. Intentional learning as a goal of instruction. In: Resnick, L. (ed.) Knowing, Learning and INstruction. Essarys in honour of Robert Glaser. Hillsdale NJ: Lawrence Erlbaum Associates.

Bergin, M. (2011). NVivo 8 and consistency in data analysis: reflecting on the use of a qualitative data analysis program.

Nurse researcher, 18(3), 6–12.

Birman, B. F., Desimone, L., Porter, A. C., & Garet, M. S. (2000). Designing professional development that works. Educational leadership, 57(8), 28–33.

Black, P., Mccormick, R., James, M. & Pedder, D. 2006. Assessment for learning and learning how to learn: a theoretical enquiry. Research Papers in Education, 21, 119-132.

Blockley, D. 2010. The importance of being process. Civil engineering and environmental systems, 27, 189-199.

Blockley, D. & Godfrey, P. 2000. Doing it Differently, systems for rethinking construction, London, Telford.

Bottery, M. 1990. The Morality of the School, London, Cassell Educational Ltd.

Bottery, M. 1992. The Ethics of Educational Management, London, Cassell Educational Ltd.

Bottery, M. 2004. The Challenges of Educational Leadership, London, Paul Chapman.

Birman, B. F., Desimone, L., Porter, A. C., & Garet, M. S. (2000). Designing professional development that works. Educational leadership, 57(8), 28–33.

Bore, A., & Wright, N. (2009). The wicked and complex in education: developing a transdisciplinary perspective for policy formulation, implementation and professional practice. Journal of Education for Teaching, 35(3), 241–256.

Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Educational researcher, 33(8),

173

3–15.

Blockley, D. 2010. The importance of being process. Civil engineering and environmental systems, 27, 189-199.

Blockley, D. & Godfrey, P. 2000. Doing it Differently, systems for rethinking construction, London, Telford.

Bronfenbrenner, U. 1989. Ecological System Theory. Annals of Child Development, 6, 187-249.

Bryk, A. S., Lee, V. E., & Peter, B. (1993). Holland, Catholic Schools and the Common Good. Also, Coulson, Market Education.

Bryk, A. S., Sebring, P. B., Allensworth, E., Easton, J. Q., & Luppescu, S. (2010). Organizing schools for improvement: Lessons from Chicago. University of Chicago Press.

Buckingham Shum & Deakin Crick, R. 2012. Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, BC: ACM Press: NY.

Cano, F. 2005. Epistemological Beliefs and approaches to learning: Their change through secondary school and their influence on academic performance. British Journal of Educational Psychology, 75, 203-221.

Checkland, P., & Scholes, J. (1999). Soft systems methodology: a 30-year retrospective. Soft systems methodology in action. Wiley Chichester.

Clarke, A., & Collins, S. (2007). Complexity science and student teacher supervision. Teaching and Teacher Education, 23(2), 160–172.

Clarke, D., & Hollingsworth, H. (2002). Elaborating a model of teacher professional growth. Teaching and teacher education, 18(8), 947–967.

Cochran-Smith, M. (2003). Learning and unlearning: The education of teacher educators. Teaching and teacher education, 19(1), 5–28.

Cohen, D. K., & Hill, H. C. (2001). Learning policy: When state education reform works. Yale University Press.

Collins, S., & Clarke, A. (2008). Activity frames and complexity thinking: Honoring both public and personal agendas in an emergent curriculum. Teaching and Teacher Education, 24(4), 1003–1014.

Crick, R. D. (2010). Integrating the personal with the public: Values, virtues and learning and the challenges of assessment. In International research handbook on values education and student wellbeing (pp. 883–896). Springer.

Crick, R. D. (2012). Deep engagement as a complex system: Identity, learning power and authentic enquiry. In Handbook of research on student engagement (pp. 675–694). Springer.

Cui, W.& Blockley, D.I. Interval Probability Theory for Evidential Support. Int.J.Intelligent Systems 5 183-192. 1990

Curtis, M. J., & Stollar, S. A. (2002). Best Practices in System-Level Change. Retrieved from http://psycnet.apa.org/psycinfo/2006-03715-016

Davies, B. (2011). Leading the strategically focused school: Success and sustainability. Sage Publications Limited.

Davis, J P & Fletcher A. Managing Assets under Uncertainty. Society of Petroleum Engineers Asia Pacific Conference on

Integrated Modelling for Asset Management. Yokohama, Japan. 2000

Darling-Hammond, L. & Bransford, J. 2005. Preparing Teachers for a Changing World: What Teachers Should Learn and be Able to Do, San Francisco, Jossey-Bass.

Davis, B., & Sumara, D. J. (2006). Complexity and education: Inquiries into learning, teaching, and research. Psychology

174

Press.

Davies, B., & West-Burnham, J. (2003). The handbook of educational leadership and management. Pearson Education.

Day, C. and Leith, R. 2007. “The continuing professional development of teachers: Issues of coherence, cohesion and effectiveness”. In International handbook of school effectiveness and improvement, Edited by: Townsend, T. 468–83. Dordrecht, , The Netherlands: Springer.

Deakin Crick, R., Jelfs, H., Ren, K., & Symonds, J. (2010). Learning futures. London: Paul Hamlyn Foundation.

Deakin Crick, R., Grushka, K., Heitmeyer, D., & Nicholson, M. (2010). Learning, Place and Identity: An exploration of the affordances of a pedagogy of place amongst Indignous Australian students. Bristol, UK: University of Bristol.

Deakin Crick, R., Jelfs, H., Huang, S. & Wang, Q. 2011. Learning Futures Final Report. London: Paul Hamlyn Foundation.

Deakin Crick, Ruth. (2009). Pedagogical challenges for personalisation: integrating the personal with the public through context-driven enquiry. The Curriculum Journal, 20(3), 185–189Deakin Crick, R. 2009. Inquiry-based learning: reconciling the personal with the public in a democratic and archaeological pedagogy. Curriculum Journal, 20, 73 - 92.

Dempster, A.P. Upper and lower probability inference for families of hypotheses with monotone density ratios. Annals

of Math. Statist. Vol.40, No.3, pp. 953-969, 1969.

Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of professional development on teachers’ instruction: Results from a three-year longitudinal study. Educational evaluation and policy analysis, 24(2), 81–112.

Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005). Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of health services research & policy, 10(1), 45–53B.

Dweck, C. S. 2000. Self-Theories: Their role in Motivation, Personality, and Development, New York, Psychology Press.

Eccles, J., Wigfield, A. & Schiefele, U. 1998. Handbook of Child Psychology,.

Fredricks, J., Blumenfeld, P. & Paris, A. 2004. School Engagement: Potential of the Concept, State of the Evidence. Review of Educational Research74, 59-109.

Freire, P. 1972. Pedagogy of the Oppressed, Harmondsworth, Penguin.

Fullarton, S. 2002. Student engagement with school: individual and school-level influences. LSAY Research Report Melbourne: Australian Council for Educational Research.

Gale, T., Tranter, D., Bills, D., Hattam, R. & Comber, B. 2008. Interventions early in school as a means to improve higher education outcomes for disadvantaged (particularly low SES) students: review of the Australian and international literature. Canberra: The Department of Education, Employment and Workplace Relations.

Guerra-Ramos, M. T., García-Horta, J. B., López-Valentín, D., Gómez-Galindo, A., & Balderas-Robledo, R. (2007). SCIENCE TEACHERS’ATTITUDES AND PERCEPCTIONS RELATED TO PRACTICAL WORK: A SELF REPORT QUESTIONNAIRE.

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American educational research journal, 38(4), 915–945.

Goldspink, C. (2007). Rethinking Educational Reform A Loosely Coupled and Complex Systems Perspective. Educational Management Administration & Leadership, 35(1), 27–50.

Goldstein, H. (1987). Multilevel models in education and social research. Oxford University Press.

Goldstone, R. 2006. The Complex Systems See-Change in Education. The Journal of the Learning Sciences, 15, 35-43.

175

Goodson, I. & Beista, G. 2010. Narrative Learning, , London and New York, Routledge.

Goodson, I. & Deakin Crick, R. 2009. Curriculum as narration: tales from the children of the colonised. Curriculum Journal, 20, 225 - 236.

Gunter, H. 2001. Leaders and Leadership in Education, London, Paul Chapman.

Grace, G. (2002). Catholic schools: Mission, markets, and morality. Routledge.

Habermas, J. (1972). Knowledge and human Interests (paperback). Beacon Press, USA. Retrieved from http://www.lavoisier.fr/livre/notice.asp?id=OLKWAXAOKA6OWO

Hall, J. K. (2004). Language Learning as an Interactional Achievement. The Modern Language Journal, 88(4), 607–612. doi:10.2307/3588591

Hall J W, Blockley D I, Davis J P Uncertain Inference Using Interval Probability Theory. Int. J. Approximate Reasoning. 19

247-264, 1998.

Hallinger, P., & Heck, R. H. (2010). Collaborative leadership and school improvement: understanding the impact on school capacity and student learning. School Leadership & Management, 30(2), 95–110. doi:10.1080/13632431003663214

Hanushek, E. & Woessmann, V. 2011. The economics of international differences in educational achievement. In: Hanushek, E., Machin, S. & Woessmann, L. (eds.) Handbook of the Economics of Education. Amsterdam: North Holland.

Hattie, J. 1999. Influences on Student Learning. Auckland: University of Auckland.

Hattie, J. 2009. Visible Learing A Synthesis of over 800 Meta-Analyses relating to learning, London, Routledge.

Hautamaki, J., Arinen, P., Eronen, S., Hautamaki, A., Kupiainen, S., Lindblom, B., Niemivirta, M., Pakaslahti, L., Rantanen, P. & Scheinin, P. 2002. Assessing Learning to Learn: a framework, Helsinki, National Board of Education.

Heron, J. & Reason, P. 1997. A Participatory Inquiry Paradigm. Qualitative Inquiry, 3, 274-294.

Harlen, W. & Deakin Crick, R. 2003a. A systematic review of the impact of summative assessment and testing on pupils' motivation for learning. London: Evidence for Policy and Practice Co-ordinating Centre Department for Education and Skills.

Harlen, W. & Deakin Crick, R. 2003b. Testing and Motivation for Learning. Assessment in Education, 10, 169-207.

Harlen, Wynne, & Crick, R. D. (2002b). Review: What is the evidence of the impact of summative assessment and tests on students’ motivation for learning.

Hidi, S. & Renninger, K. A. 2006. The Four-phase Model of Interest Development. Educational Psychologist, 41, 111-127.

Hofer, B. K. 2001. Personal Epistemology Research: Implications for Learning and Teaching. Journal of Educational Psychology Review, 13, 353-383.

Jackson, M. C. (2000). Systems approaches to management. Springer. James, M., & Gipps, C. (1998). Broadening the basis of assessment to prevent the narrowing of learning. Curriculum journal, 9(3), 285–297.

Jackson, D., & Temperley, J. (2007). From professional learning community to networked learning community. Professional learning communities: Divergence, depth and dilemmas, 45–62.

James, M. & Gipps, C. 1998. Broadening the basis of assessment to prevent the narrowing of learning. The Curriculum Journal, 9, 285-297.

James, M., Mccormick, R., Black, P., Carmichael, P., Drummond, M., Fox, A., Macbeath, J., Marshall, B., Pedder, D.,

176

Procter, R., Swaffield, S., Swann, J. & Wiliam, D. 2007. Improving Learning How to Learn: Classrooms, Schools and Networks, London, Routledge.

Katz, S., & Earl, L. (2010). Learning about networked learning communities. School Effectiveness and School Improvement, 21(1), 27–51.

Kyriakides, L., & Campbell, R. J. (2004). School self-evaluation and school improvement: A critique of values and procedures. Studies in educational evaluation, 30(1), 23–36.

Ladwig, J., Smith, M., Gore, J. M., Amosa, W. & Griffiths, T. 2007. Quality of Pedagogy and Student Acheivement: Multi-level replication of Authentic Pedagogy. ANARE Annual Conference. Freemantle, Western Australia.

Leithwood, K. & Jantzi, D. 1999. The relative effects of principal and teacher sources of leadership on student engagement with school. Educational Administration Quarterly, 35, 27.

Little, A. D., & Veugelers, R. (2005). Internationalisation of R&D in the UK. Report to the office of Science and Technology.

Loxley, A., Johnston, K., Murchan, D., Fitzgerald, H., & Quinn, M. (2007). The role of whole-school contexts in shaping the experiences and outcomes associated with professional development. Journal of In-Service Education, 33(3), 265–285.

Mahn, H. & John-Steiner, V. 2002. Developing the affective zone of proximal development. In: Wells, G. & Claxton, C. (eds.) Learning for Life in the 21st Century, Socio-cultural perspectives on the future of education. Oxford: Blackwell.

Macbeath, J., & CHENG, Y. C. (2008). Leadership for learning: International perspectives. Retrieved from http://repository.ied.edu.hk/dspace/handle/2260.2/8940

MacBeath, J., & McGlynn, A. (2002). Self-evaluation: What’s in it for Schools? Routledge.

Marshall, B. & Drummond, M. 2006. How teachers engage with Assessment for Learning: lessons from the classroom. Research papers in Education, 21, 133-149.

Mason, M. (2008). What is complexity theory and what are its implications for educational change? Educational Philosophy and Theory, 40(1), 35–49.

Mason, M. (ed.) 2008. Complexity Theory and the Philosophy of Education, Chichester: Wiley-Blackwell.

McCombs, B. L., & Whisler, J. S. (1997). The Learner-Centered Classroom and School: Strategies for Increasing Student Motivation and Achievement. The Jossey-Bass Education Series. ERIC.

Mckinsey Report 2007. How the world's best performing school systems come out on top. London: McKInsey.

Munns, G. & Mcfadden, M. 2000. First Chance, Second Chance or Last Chance? Resistence and Response to education. British Journal of Sociology of Education, 21, 59-75.

Newmann, F. & Wehlage, G. 1995. Successful school restructuring, Madison WI, Center on Organisation and Restructuring of Schools.

Opfer, V. & Pedder, D. 2011. The lost promise of teacher professional development in England. European Journal of Teacher Education, 34, 3-24.

Meiers, M., & Ingvarson, L. (2005). Investigating the links between teacher professional development and student learning outcomes. Department of Education, Science and Training. Mingers, J., & Brocklesby, J. (1997). Multimethodology: towards a framework for mixing methodologies. Omega, 25(5), 489–509.

Morgan, G., & Smircich, L. (1980). The case for qualitative research. Academy of management review, 5(4), 491–500.

177

Muijs, D. (2010). Doing quantitative research in education with SPSS. Sage.

PriceWaterhouseCoopers. 2007. The economic benefits of a degree. Research report. London: Universities UK.

Pedder, D. 2006. Organizational conditions that foster successful classroom promotion of learning how to learn. Research Papers in Education,, 21, 171-200.

Pedder, D. 2007. Profiling teachers' professional learning practices and values: differences between and within schools. The Curriculum Journal 18, 231-252.

Pedder, D., James, M. & Macbeath, J. 2005. How teachers value and practise professional learning. Research Papers in Education, 20, 209-243.

Pedder, D., & MacBeath, J. (2008). Organisational learning approaches to school leadership and management: teachers’ values and perceptions of practice. School Effectiveness and School Improvement, 19(2), 207–224.

Piesanen, E., Kiviniemi, U., & Valkonen, S. (2007). Follow-up and Evaluation of the Teacher Education Development Programme. Continuing Teacher Education in 2005 and its Follow-up 1998-2005 by Fields and Teaching Subjects in Different Types of Educational Institutions. Occasional papers, 38.

Polanyi, M. (1967). The tacit dimension. 1967. Garden City, New York, 217.

Reay, D., & Wiliam, D. (1999). ’I’ll be a nothing’: structure, agency and the construction of identity through assessment [1]. British Educational Research Journal, 25(3), 343–354.

Ren, K., & Deakin Crick, R. (2013). Empowering underachieving adolescents: an emancipatory learning perspective on underachievement. Pedagogies: An International Journal, (ahead-of-print), 1–20.

Ritchie, R. (2007). School Self-Evaluation. Advances in Program Evaluation, 10, 85–101. doi:10.1016/S1474-7863(07)10006-5

Robinson, V. M., Lloyd, C. A., & Rowe, K. J. (2008). The impact of leadership on student outcomes: An analysis of the differential effects of leadership types. Educational administration quarterly, 44(5), 635–674.

Rosenhead, J., & Mingers, J. (2001). Rational analysis for a problematic world revisited. John Wiley and Sons.

Sammons, P. (1999). School effectiveness: Coming of age in the twenty-first century (Vol. 6). Taylor & Francis.

Seely Brown, J. & Thomas, D. 2009. Learning for a World of Constant Change. 7th Gillon Colloquium. University of Southern California.

Sfard, A. & Prusak, A. 2005. Telling Identities: In Search of an Analytic Tool for Investigating Learning as a Culturally Shaped Activity. Educaitonal Researcher 34, 14-22.

Shernoff, D. J., Csikszentmihalyi, M., Schneider, B. & Steele , E. 2003. Student Engagement in High School classrooms from the perspective of flow. . School Psychology Quarterly, 18, 18.

Silins, H. M., Bill 2001. Reframing Schools: The Case for system, teacher and student learning. AARE.

Shafer, G. A Mathematical Theory of Evidence, Princetown: Princetown University Press, 1976.

Shulman, L. (2011). The skills of helping individuals, families, groups, and communities. Cengage Learning.

Shum, S. B., & Crick, R. D. (2012). Learning dispositions and transferable competencies: pedagogy, modelling and learning analytics. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 92–101).

Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of educational psychology, 82(3), 498.

178

Stoll, L., & Louis, K. S. (2007). Professional learning communities: Divergence, depth and dilemmas. McGraw-Hill International.

Sykes, G. (1996). Reform OF and AS Professional Development. Phi Delta Kappan, 77(7), 464–67.

Teddlie, Charles, & Reynolds, D. (2000). The International Handbook of School Effectiveness Research. Routledge.

Timperley, H., & Alton-Lee, A. (2008). Reframing teacher professional learning: An alternative policy approach to strengthening valued outcomes for diverse learners. Review of research in education, 32(1), 328–369.

Wheatley, M. 1999. Leadership and the new science: Discovering order in a chaotic world (2nd ed.), San Francisco, Berrett-Koehler Publishers.

Wheatley, M. & Kellner-Rogers, M. 1998. Bringing Life to Organizational Change. Journal for Stategic Performance Management, April/May.

Wheatley, M. 1999. Leadership and the new science: Discovering order in a chaotic world (2nd ed.), San Francisco, Berrett-Koehler Publishers.

Wheatley, M. & Kellner-Rogers, M. 1998. Bringing Life to Organizational Change. Journal for Stategic Performance Management, April/May.

Willms, J. 2003. Student engagement at school: a sense of belonging and participation: results from PISA 2000. Paris: OECD.

Zevenbergen, R., Mousley, J. & Sullivan, P. 2004. Making the pedagogic relay inclusive for indigenous Australian students in mathematics classrooms. International Journal of Inclusive Education 8, 391-405.

Zohar, D. 1997. Rewiring the Corporate Brain: using the new science to rethink how we structure and lead organisations, San Francisco, Berrett-Koehler.

Zyngier, D. 2004. Doing Education not Doing Time: Engaging Paedagogies and Pedagogues - what does student engagement look like in action? International Education Research Conference Melbourne.

Weaver, W. (1948). Science and complexity. American scientist, 36(4), 536–544.

Woolfolk Hoy, A., Hoy, W. K., & Davis, H. A. (2009). Teachers’ self-efficacy beliefs. Handbook of motivation at school, 627–654.

Veenman, M. V., Van Hout-Wolters, B. H., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and learning, 1(1), 3–14.

Vygotsky, L. 1978. Mind and society: the development of higher psychological processes, MA, Harvard University Press.

179

APPENDIX

APPENDIX 1 GLOSSARY OF TERMS

Research into the properties of social systems suggests that there are some key properties which are particularly relevant in understanding schools. These provide the project with key reference points for testing, critique and revision as we learn to what extent they help researchers and school leaders make sense of the dilemmas they face.

Autopoiesis: the process whereby a system reproduces itself. System components generate recursively the same network of processes which produced them - this is relevant to the emergence of learner identity in co-constructed domains of meaning (De Jaegher and Di Paolo, 2007; Kay, 2002; Goldspink and Kay, 2009; Goldspink, 2009) and hence for the way we approach learning as well as school change.

Dissonance: lack of agreement, alignment, balance or conflict between people, processes or things - this creates a space for deep learning through identifying and formulating problems, and finding solutions, or resolutions, as a key element of enquiry.

Emergence: the way complex systems and patterns arise out of a multiplicity of relatively simple interactions - the creation of something new from the relationships between processes is a space for creativity and learning – creative leadership is required to manage complex organisations (Sawyer, 2001; Goldspink, 2010; Stacey, 1996).

Resilience: the ability of a system or person to continue functioning when stressed and where necessary, adapt and change in response to challenge - this is a core aspect of learning power - for individuals, organisations and communities.

Synergy: the whole is more than the sum of its parts - a learning community has a culture and synergy which cannot be reduced to a single process. In addition to these we include trust.

Trust: relationships of such quality that they can withstand the challenges of risk, uncertainty, inequality and difference (Bryk et al 2010; Deakin Crick 2007; Bond 2004) and identity.

Identity: learning that enables the identification and location of ‘self’ in personal, organisational and community narratives (Varela, 1997; Bruner, 1990, 1996; Sfaard & Pusak 2005; Deakin Crick, 2011).

APPENDIX 2 OASIS CHARTER

Oasis Community Learning’s Education Charter was agreed by the Board of Trustees in November 2009. It is based on the views of all the main stakeholders in Oasis Academies and serves as a statement of the organisation’s overall values, vision and purposes. At the time, the Trustees suggested that wider measures of the performance of Oasis Academies might be developed, hence the ECHO Project. Between 2010 and 2013, ECHO has focused on a pilot project to evaluate the

180

‘Transforming Learning’ section of the Charter. The other two sections are ‘Transforming Lives’ and ‘Transforming Communities’.

TRANSFORMING LEARNING - We have a passion for learning and we want everyone to achieve

their full, God-given potential

Creating environments in all our Academies where learning is fun

Providing all our students with excellent opportunities for learning and assessment to support learning

Ensuring that students maintain good progress between the primary and secondary phases and other key transitions

Aiming for ‘good’ to ‘outstanding’ standards of learning and teaching in all our Academies and, where necessary, getting the basics right to raise standards of attainment quickly

Understanding and thinking critically about different worldviews as frameworks for learning

Providing a forward-looking, broad and balanced curriculum which develops the gifts and talents of all students and those that teach and lead them

Ensuring that Oasis Academies become centres of excellence for their specialisms

Ensuring that students are accessing a wide range of extended opportunities for learning within and beyond the Academy

Ensuring that learning balances knowledge and understanding with skills for life and caters for the whole person: academically, vocationally, socially, morally, spiritually, physically, emotionally and environmentally

Providing progression routes for all students in the secondary phase that are attractive and accessible, helping them to meet challenging personal targets and achieve excellent results

Enabling students to make a successful transition to interdependent living through employment, further/higher education or other beneficial activities; becoming confident, competent and fulfilled adults

Providing learning and teaching in an environment that is healthy and safe and where there are excellent services for care, guidance and support, particularly for the most vulnerable students

Developing a culture of mutual accountability and rigorous evaluation; being well prepared for external review and inspection

Managing the resources that we are given in a responsible way to ensure maximum impact on the quality of education provided

Developing our staff and sharing our successes between the Oasis Academies and more widely

Measuring performance based on the aspirations and outcomes of the Education Charter; committing to continuous improvement

Celebrating success widely and regularly; supporting others with their learning and being pleased when others do well

181

APPENDIX 3 CRITICAL SUCCESS INDICATORS (CSI)

The ECHO Project will focus on CSIs relating to the learning outcomes of the Education Charter,

particular the section ‘Transforming Learning’. This will complement the Academy/Hub evaluation

projects that focus on spiritual, moral, social and cultural development and on the Oasis ethos.

The sets of questions that form the basis of the evaluation strands will be designed to provide

evidence for one or more of the twelve 3rd Layer CSIs. The subsequent Perimeta analysis will

provide developmental feedback to the Academies about quality of performance on the CSIs

according to the colours of the Italian flag: red = not being achieved; green = being achieved; white

= insufficient evidence to make a judgement.

Critical Success Indicators for an Oasis Academy

1st Level CSI:

1.1 Establishing and sustaining a group of high achieving learning communities that enables

everyone to realise their full potential and refuses to put limits on achievement

2nd Level CSIs:

2.1 Developing the learning of students so that they realise their full potential

2.2 Developing the learning of teachers so that they realise their full potential

2.3 Developing the learning of leaders so that they lead the learning of teachers and students effectively

2.4 Engaging parents/carers effectively in the learning activities of the Academy and in supporting the learning of their children

3rd Level CSIs:

3.1 Feeling valued and included in our community (evidence from students & staff)

3.2 Feeling part of a community that focuses its activities on learning (students & staff)

3.3 Knowing that we are being helped to reach our full potential and achieve our personal best (students & staff)

3.4 Maintaining a good rate of progress at key transitions, for example, primary to secondary school (students)

3.5 Achieving results that meet my best expectations (students & staff)

3.6 Feeling that my successes are recognised (students & staff)

182

3.7 Contributing actively to a community that focuses on service to others (students & staff) 3.8 Contributing actively to a community where shared leadership is promoted (students & staff)

3.9 Providing evidence that training and development opportunities have helped me to improve my

classroom practice (staff)

3.10 Contributing actively to a community that learns together, shares what works best and knows what it needs to do to improve (staff)

3.11 Feeling included and involved with my children’s education at the Academy and knowing that there is an open door for contact when I need it (parents/carers)

3.12 Being effective role models as leaders of learning (senior/middle leaders)

APPENDIX 4 QUESTIONAIRES

STRAND ONE – STUDENTS SELF-REPORT QUESTIONNAIRE ON LEARNING POWER

A 72 item research validated online questionnaire, the Effective Lifelong Learning Inventory (ELLI).

STRAND TWO - STUDENTS PERCEPTIONS OF EDUCATION CHARTER

Student Questionnaire

This questionnaire will help us to find out more about what our students think and feel about their

learning, teaching, progress and achievement in an Oasis Academy. By getting your views we can

celebrate what is going well and try to make improvements where things are not going as well as

you would like.

When we analyse the results of the questionnaire, we shall be combining all the students’

responses and so what you say as an individual student will be confidential.

Please answer the questions by marking the response that best describes what you think or feel

about each statement.

Questions

1. I feel included in the activities that are available to me in the Academy

*Strongly agree *Agree *Disagree *Strongly disagree

(Repeat after each question)

2. I feel safe in the Academy

3. I feel valued and cared for as an individual in the Academy

4. I like learning new things

5. I usually concentrate on my learning

183

6. I am hard to distract

7. I like working with other students to help my learning

8. I don’t distract other students

9. I am confident in my learning

10. I take a lot of care with my work

11. I work hard

12. I do more than I am asked to do in class

13. I don’t mind making mistakes because I learn from them

14. I come up with new ideas to help my learning

15. I enjoy my learning

16. My teachers teach well so that I learn successfully

17. I have regular opportunities to express my opinion to my teachers about my learning

18. I often have conversations with my teachers which help me to make better progress

19. My teachers seem to enjoy teaching my classes

20. My teachers often share things that they have learned themselves

21. I get on well with my teachers

22. I get extra support to help my learning when I need it

23. I know that the senior teachers want us to learn successfully

24. Everyone works very hard to make the Academy a great place to learn and to do well

25. My teachers constantly expect me to improve on my personal best

26. Right now I am achieving the best I possibly can

27. From what I remember, I kept up a good rate of progress between my last year at primary

school and my first year at the Academy

28. I have kept up a good rate of progress ever since my first year at the Academy

29. I think that my results at the moment are as good as I can do

30. I feel that my successes at the Academy are recognised

31. I feel that my successes with activities outside the Academy are recognised

32. I feel proud about what I have achieved so far

33. We are taught about the responsibilities of students as well as their rights

34. We care about helping each other in the Academy community

35. We are encouraged to care about the needs of other people in the local community and

around the world

36. I regularly offer to help others

37. My parents or carers feel that they are always welcome in the Academy

38. My parents or carers feel involved with my learning and my progress

39. My parents or carers are pleased with my progress

40. My parents or carers have always been able to sort out any problems that I have had

41. I am encouraged to lead activities in the classroom

42. I have had opportunities to lead other activities at the Academy

(For example, sports teams, fund-raising, student council activities)

184

STRAND THREE – STUDENT NARRATIVE INTERVIEWS

Student Interview Schedule

This is a semi-structured interview schedule, designed to invite students to tell stories about their

learning. We are interested in how they understand and experience success and struggle in

learning, how they understand themselves as learners and whether they see themselves as

changing and improving over time. These questions are not designed to be read verbatim – rather

they provide a structured framework for a natural conversation. If a question doesn’t work –

reframe it, and feel free to add open-ended prompts.

It is important that you tell the students why we are asking these questions and what will happen

to the recording we make. Please see the ECHO Consent Guidance and stress that:

The data that we gather will be anonymous and not traceable to a particular student.

There is no right answer to each question.

We want to know how people are learning in our Academy so we can improve what we do

and make it a better place to learn.

At the start of each recording, please state the student’s unique reference number.

Questions

8. Tell me about a time when you learned something really successfully in the last two or three

weeks

Why did you learn successfully?

What did it feel like

What happened as a result?

9. Tell me about your worst experience of learning in the last two or three weeks?

Why was it a bad experience of learning?

What did it feel like?

What happened as a result?

10. How different are you now as a learner compared with a year ago?

How would you describe the differences?

What has happened to make these differences?

How do you feel now about the future as a learner?

What changes would help you to learn better?

11. How would you describe yourself as a learner now? If you could choose an animal (or a car)

which is most like you as a learner, which animal (or car) would it be? Why did you choose

that particular animal (or car)? [Probe further for reasons, if necessary]

12. What have been the best things about your time at the Academy so far?

13. What have been the worst things about your time at the Academy so far?

14. How do you feel about your future, after your time at the Academy?

185

STRAND FOUR – YEAR 11 LEAVERS PERCEPTIONS OF THEIR EXPERIENCE AT THEIR OASIS

ACADEMY

1. I was valued and included during my time at the Academy

*Strongly agree *Agree *Disagree *Strongly disagree

(Repeat after each question)

2. I felt part of a community that was focused on learning

3. I felt safe at the Academy 4. 5.

6. I enjoyed my time at the Academy 7.

8. I did as well as I had hoped in my GCSEs 9.

10. My teachers accurately predicted my GCSE results

11. I am looking forward to the move to Sixth Form, Further Education, training or work

12. I have been well prepared for the next stage of my life

13. My successes were recognised during my time at the Academy

14. I was helped and supported whenever I struggled with my learning

15. I developed a sense of responsibility towards other people during my time at the

Academy

16. The Academy helped to make me more understanding about different cultures and

types of people

17. I supported activities that would improve things for other people

18. I was offered opportunities for leadership during my time at the Academy

19. I led activities during my time at the Academy

20. My teachers taught me well at the Academy 21.

22. My teachers were always trying to improve their teaching

23. I was regularly challenged to do better so that I could reach my full potential

24. My parents/carers felt involved with my education at the Academy

25. My parents felt that they could find the right person to talk to at the Academy if they

had any concerns about my education

26. Senior staff at the Academy wanted us to learn well and achieve our best

27. Everyone at the Academy worked very hard to make it a great place to learn and to do

well

28. The Academy was well run 29. 30.

31. There were great facilities for learning 32.

33. There was a broad curriculum that helped me to develop as a whole person

34. I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum

35. The Academy made good use of new technologies for learning

36. I never felt that anyone was putting limits on what I could achieve

37. I feel proud about what I achieved at the Academy

186

38. I could not have gone to a better place for my secondary education

STRAND FIVE - PERCEPTIONS OF PARENTS AND CARERS OF THEIR CHILDREN'S

ACADEMIES

This questionnaire has been designed to gather evidence about parent/carer experiences of the

education of their children at the Academy. Your views will help us to plan further improvements

that will provide an even better education for all our students.

Please answer the questions in Part I by marking the most appropriate response for each question

based on the experience of your child at the Academy. If you have more than one child at the

Academy and their experiences have been different, please complete an additional questionnaire.

Part II provides some additional questions for parents/carers in families where the first language is

not English.

Your responses will be treated with complete confidence. We shall only be reporting on the

combined views of all the parents/carers who complete the questionnaire. If we do want to follow

up individual responses, we shall ask for your permission.

Thank you for completing the questionnaire.

Part I: General Questions

1. My child is known and valued as an individual

*Strongly agree *Agree *Disagree *Strongly disagree

2. The Academy is providing for my child’s particular needs

3. My child feels included in the life of the Academy, in classrooms and in other activities

4. The Academy focuses its work on providing the best opportunities for learning for all

students

5. My child is taught well

6. My child is being challenged to improve on previous best performance in all areas

7. My child’s progress at the Academy has exceeded our expectations

8. My child’s successes are recognised

9. The Academy ensures my child is well looked after

10. The Academy helps my child to have a healthy lifestyle

11. There is a good standard of behaviour at the Academy

12. My child is developing an attitude of helping others

(for example, by doing practical activities to improve life at the Academy, by supporting

people in need, getting involved with community projects, fund-raising)

13. My child is taking opportunities to lead activities (for example, leading group work in the

classroom, leading a sports team, giving feedback to teachers about their learning, being a

representative on a Student Council, taking a lead role in Academy performances of music,

drama or dance, being a coach, mentor or ‘buddy’)

187

14. My child has kept up a good rate of progress at key transition (for example, from primary to

secondary school; from Year 9 to GCSE courses in Years 10 & 11; from Year 11 to Sixth

Form/College; from Year 13 to further education/higher education/employment)

15. I get regular and helpful information about my child’s progress

16. The Academy responds well to my concerns

17. I always find staff knowledgeable about my child

18. I always find staff helpful when discussing my child’s progress

19. The Academy helps me to support my child’s learning

20. The Academy is led and managed effectively

21. My impression is that senior staff focus their work on making the Academy a good place to

learn

22. My impression is that senior staff expect all students to achieve well

23. My child enjoys their education at the Academy

24. I would recommend the Academy to another parent/carer

25. If you want to explain any of your answers, or if there is anything else you want us to be

aware of about your child’s experience of the Academy, please give details here:

Part II: Additional questions for parents/carers whose first language is not English

26. The Academy has been helpful in ensuring that I understand the general information sent to

parents/carers

27. The Academy has been helpful in ensuring that I understand the information about my

child’s progress

28. I feel that the Academy understands and takes proper account of the particular issues and

sensitivities of my culture

29. If I have a concern, I feel confident that I can make contact with relevant staff

30. If I have a concern, I feel confident that I can communicate with staff effectively

31. I would recommend the Academy to another parent/carer from my community

32. If you want to explain any of your answers, or if there is anything else you want us to be

aware of about your child’s experience of the Academy, please give details here:

188

STRAND SIX - LEARNING OF TEACHERS: IMPACT OF CPD ON THE QUALITY OF

CLASSROOM PRACTICE

Teacher Questionnaire This questionnaire is based on two sets of research-validated questions about teachers’ learning from the national Learning How to Learn project, based at the Department of Education, Cambridge University, (Parts A and B). We are grateful for permission to use these questions. There are six additional questions in a final section that focus on aspects of the Oasis Education Charter, (Part C). Your responses will be treated with complete confidence. We shall only be reporting on the combined views of all the teachers completing the questionnaire and the teacher interview. If we do want to follow up individual responses, we shall ask your permission beforehand. We are very grateful for the time that you have given to these aspects of the ECHO Project. Please complete the questionnaire by marking the most appropriate response to each question based on your experience as a teacher in this academy.

PART A – PROFESSIONAL LEARNING

1. Staff as well as students learn in this academy

* Very often *Often *Sometimes *Rarely 2. I draw on good practice from other schools as a means to further my own professional

development 3. I read research reports as one source of useful ideas for improving my practice 4. I use the web as one source of useful ideas for improving my practice 5. I consult my students about how they learn most effectively 6. I relate what works in my own practice to research findings 7. I am able to see how practices that work in one context might be adapted to other contexts 8. I use insights from my own professional learning to feed into academy policy development 9. I reflect on my practice as a way of identifying my professional learning needs 10. I experiment with my practice as a conscious strategy for improving classroom teaching and

learning 11. I modify my practice in the light of feedback from my students 12. I modify my practice in the light of published research evidence 13. I modify my practice in the light of evidence from self-evaluations of my classroom practice

189

14. I modify my practice in the light of evidence from evaluations of my classroom practice by managers or other colleagues

15. I carry out joint research/evaluation with one or more colleagues as a way of improving my practice

16. I collaborate to plan my teaching 17. I observe other colleagues in the classroom and we give each other feedback 18. I engage in team teaching as a way of improving practice 19. If I have a problem with my teaching I turn to colleagues for help 20. I suggest ideas or approaches for colleagues to try in class 21. I make collective agreements to test out new ideas 22. I discuss openly with colleagues what and how they are learning 23. I use informal opportunities to discuss how children learn 24. I offer reassurance and support to colleagues 25. My practice is based on the belief that all students are capable of learning 26. My experience is that students in this academy enjoy learning 27. My experience is that student success is celebrated 28. I discuss with colleagues how students might be helped to learn how to learn

PART B – ACADEMY LEADERSHIP PRACTICES AND SYSTEMS

1. Senior leaders communicate a clear vision of where the academy is going

*Very often *Often *Sometimes *Rarely 2. Staff have a commitment to the whole academy as well as to their department, key stage

and/or year group 3. Senior leaders promote commitment among staff to the whole academy as well as to the

department, key stage and/or year group 4. There is effective communication between senior leaders and teachers 5. There are processes for involving all staff in decision-making 6. Teachers’ professional know-how is used in the formulation of academy policy and goals 7. Teachers’ professional know-how is used in the formulation of academy policy, even where

this leads to a questioning of established rules, procedures and practices 8. Opportunities are provided for teachers to critically evaluate academy policy 9. Staff are involved in evaluating academy policy 10. Staff participate in important decision-making 11. There are processes for involving students in decision-making 12. Staff have a good working knowledge of the Academy Development Plan (or your academy’s

equivalent document) 13. Staff see the Academy Development Plan as relevant and useful to learning and teaching 14. Staff development time is used effectively to realise Academy Development Plan priorities 15. Staff development time is used effectively in the academy 16. The academy provides cover to allow staff joint planning time 17. Teachers are encouraged to experiment with new ideas as a way of promoting professional

growth 18. Formal training provides opportunities for teachers to develop professionally 19. Teachers are helped to develop skills to assess students’ work in ways that move students

on in their learning

190

20. Teachers are helped to develop skills to observe learning as it happens in the classroom 21. Senior leaders support teachers in sharing practice with other schools through networking 22. Information is collected from teachers on those aspects of their work that they themselves

think they do effectively 23. Information is collected from teachers on effective ways they promote learning to learn

skills and knowledge among their students 24. Information is collected from teachers on informal teacher networking in which they play an

active role 25. Teacher-initiated networking is an integral element of staff development 26. Learning how to learn is an issue discussed in staff development time

PART C – ASPECTS OF THE OASIS EDUCATION CHARTER

1. I feel included in a community that values what I can offer to the staff team and to students

*Very often *Often *Sometimes *Rarely 2. I believe that I am working in a community that focuses our attention and activities on

learning to improve teaching and student outcomes 3. My experience suggests that other members of staff are involved in training and

development activities to improve their classroom practice and student outcomes 4. I have had opportunities to work alongside and/or train with colleagues in other phases to

improve students’ progress, e.g. secondary with primary teachers or secondary with tertiary phase teachers

5. My experience is that senior leaders are role models as ‘leaders of learning’ in the academy 6. Leadership is shared in this academy and opportunities to take on leadership roles are

available to all those that want them and have the relevant knowledge and experience 7. My experience is that the academy promotes a balance between the rights and the

responsibilities of all members of the community 8. There is an emphasis among students and staff on caring about and helping other people in

the academy community 9. There is an emphasis among students and staff on caring about and helping other people in

the wider community 10. The academy is successful in helping parents/carers to support the learning of their children 11. The achievements of staff are recognised and celebrated 12. I believe that I am realising my full potential as a teacher

PART D – BACKGROUND INFORMATION

Please tick or fill in the appropriate boxes: 1. Female --- Male --- 2. Name of your academy -------------------------------- 3. Years of teaching experience --- 4. Years at this academy --- 5. Post and responsibility (if relevant) --------------------------------- 6. Please identify your main area of responsibility ----------------------------- 7. Please identify the main subjects that you teach -----------------------------

191

STRAND SEVEN - SENIOR LEADERSHIP

Individual questionnaire for senior leaders, followed by a team discussion

Introduction

This questionnaire is part of the ECHO Project which is exploring approaches to broaden the range

of evidence used to evaluate Oasis Academies. The Project, based on a partnership between Oasis

Community Learning, University of Bristol and PA Consulting, is being undertaken as a pilot study in

three Oasis Academies. We are using Perimeta software to analyse the data generated by the

evidence-gathering processes. It is hoped that the analysis will then help to inform strategic

thinking and development planning in each Academy, so that student outcomes continue to

improve across a broader range of objectives.

The ECHO Project is focusing on evaluation of the ‘Transforming Learning’ dimension of OCL’s

Education Charter, (ECHO stands for Evaluating Charter Outcomes). We are using a model that

proposes the effective relationship between student learning, teacher learning, the leadership of

learning and parental support as a key to creating Oasis Academies as high-achieving learning

communities. Our aim is to develop processes for evaluating the broader aspects of education in

Oasis Academies that are both affordable and sustainable.

For the pilot project in 2011/12, we shall only be investigating the role of senior leaders in the

leadership of learning. If the pilot is successful, it will be possible to extend this aspect of evidence-

gathering to other leaders in subsequent years.

In devising this questionnaire, we have drawn from the following international sources about the

relationship between leadership, learning, teaching, student attainment and overall school

improvement, including several research-validated questionnaires:

Questionnaires from the Learning How to Learn research project (2002, University of

Cambridge)

Resources for Learning-centred Leadership (2004, National College)

Framework for Teaching for Effective Learning: Unleashing learning potential (2010,

Government of South Australia)

Essential supports for improvement from Organizing Schools for Improvement: Lessons from

Chicago (2010, Anthony S Bryk, et al., University of Chicago Press)

Core behaviours from Resourceful leadership: how directors of children’s services improve

outcomes for children (2011, National College)

The questionnaire will be completed by individual members of the senior team at your Academy

and returned to Howard Green a few days before a meeting of the Academy Leadership Team. The

192

aggregated responses will then be used to identify common threads, anomalies and emerging

issues and to create an agenda for discussion at the ALT meeting. The confidentiality of individual

responses is guaranteed. On this basis, please would you put your name in the space below:

Your name:

Leadership of Learning Questionnaire

Please answer the following questions as honestly and objectively as possible, either underlining

the appropriate option on the four-point scale or (for the last question only) by providing a brief

written response. If you are not a teacher/teaching member of the senior leadership team, please

leave the questions that are not relevant (for example, numbers 2 and 3).

1. I have a good understanding of my own and other people’s learning

*Strongly agree *Agree *Disagree *Strongly disagree

2. I keep my knowledge about teaching and learning and about my phase and/or subject specialism up to date

3. I teach regularly and my lessons are rated good or outstanding

4. I undertake regular and substantial professional development for myself

5. I understand the differences between good and outstanding lessons

6. I regularly work with staff to improve their teaching practice

7. I can design, monitor and assess programmes for effective teaching and learning

8. I regularly participate in critical and reflective enquiry to develop teaching and learning across the academy

9. I understand what an academy needs to do to become good and outstanding

10. I regularly work with staff to help them improve their leadership and management

11. I am an effective coach and use my coaching skills regularly to help others to improve their performance

12. I regularly interact with staff in other schools, academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our academy

13. I make regular and effective contributions to educational dialogue that shapes whole academy policy and informs practice

14. I contribute effectively to the critical development of academy systems and structures in order to ensure quality teaching focused on student progress in learning

193

15. The senior leadership team communicates a clear vision of where the academy is going so that there is a shared sense of purpose

16. Staff have a commitment to the whole academy as well as to their department, key stage and/or year group

17. The senior leadership team effectively promotes commitment among staff to the future development of the whole academy

18. There is effective communication between senior leaders and teachers

19. There are processes for involving all staff in decision-making

20. Teachers’ professional know-how is used in the formulation of academy policy and goals

21. Teachers’ professional know-how is used in the formulation of academy policy, even where this leads to a questioning of established rules, procedures and practices

22. Opportunities are provided for teachers to critically evaluate academy policy

23. Staff participate in important decision-making

24. There are processes for involving students in decision-making, including how best to improve classroom practice

25. Staff have a good working knowledge of the Academy Development Plan

26. Staff see the Academy Development Plan as relevant and useful to learning and teaching

27. Staff development time is used effectively to realise Academy Development Plan priorities

28. Staff development time is used effectively in the academy, particularly through the explicit modelling of good practice

29. The academy provides support to allow staff joint planning time

30. Teachers are encouraged to experiment with new ideas as a way of promoting professional growth and improving classroom practice

31. Formal training provides opportunities for teachers to develop professionally

32. Teachers are helped to develop skills to assess students’ work in ways that move their students on in their learning

33. Teachers are helped to develop skills to observe learning as it happens in the classroom

34. Senior leaders support teachers in sharing practice with other schools and academies through networking

194

35. Information is collected from teachers on those aspects of their work that they themselves think they do effectively

36. Information is collected from teachers on effective ways they promote ‘learning tolearn’ skills and knowledge among their students

37. Information is collected from teachers on informal teacher networking in which they play an active role

38. Teacher-initiated networking is an integral element of staff development

39. ‘Learning how to learn’ is an issue discussed in staff development time

40. There is a well-developed shared language and a rich dialogue about teaching and learning across the academy

41. We are good at systematic monitoring, use of data and feedback across the academy

42. We have efficient and effective support systems and structures across the academy

43. The senior leadership team are effective role models as ‘leaders of learning’, (for example, by securing broad commitment to learning for all that is focused on improving the quality of teaching and student outcomes)

44. We have developed successful approaches to distributed leadership across the academy where the pathways of accountability are working well

45. The academy is successful in helping parents/carers to support the learning of their children

46. The governors play a full and appropriate part in strategic planning to improve teaching and learning and student outcomes

47. The academy involves a wide range of other stakeholders directly and powerfully in the education of students

48. The academy is good at securing the broadest possible impact of successful innovations in teaching and learning

49. There is a widespread and well-established climate of trust in the academy

50. There is an appetite to face weaknesses and bring about changes for improvement in the academy

51. Good personal relationships and strong teams help us to face challenges with confidence

52. The senior leadership team has the skills and resilience to successfully work through the more complex challenges that we are facing

53. The senior leadership team is good at long term planning for capacity building (for example, successfully addressing issues like student progress across key transitions; improving the ‘tail’ of student underachievement and/or significant achievement gaps; succession planning)

195

54. As an academy, we are good at celebrating success

55. If there is one thing that you think would improve the leadership of learning in the academy, what would it be?

APPENDIX 5 FULL PERIMETA RESULTS

TABLE 146 STRAND 1 FULL RESULTS

Male students

Q Statement Time 1 Time 2

POS UNC NEG POS UNC NE

G

The learning of students - development of students as learners in Yr 7 0.53 0.30 0.17 0.80 0.16 0.04

ACADEMY 1 Enfield 0.32 0.39 0.28 0.87 0.11 0.02

ACADEMY 2 Lords Hill 0.24 0.32 0.44 0.80 0.17 0.03

ACADEMY 3 Wintringham 0.53 0.30 0.17 0.80 0.16 0.04

GENDER 1 Male 0.53 0.30 0.17 0.80 0.16 0.04

GENDER 2 Female – not included

S1 1 Changing & Learning 0.61 0.26 0.13 0.60 0.27 0.13

S1 2 Learning Relationships 0.49 0.31 0.20 0.56 0.29 0.15

S1 3 Strategic Awareness 0.40 0.32 0.28 0.74 0.20 0.06

S1 4 Resilience 0.41 0.33 0.27 0.34 0.39 0.26

S1 5 Creativity 0.38 0.33 0.29 0.70 0.22 0.08

S1 6 Meaning Making 0.67 0.24 0.08 0.57 0.28 0.15

S1 7 Critical Curiosity 0.53 0.30 0.17 0.80 0.16 0.04

Female students

Q Statement Time 1 Time 2

POS UNC NEG POS UN

C

NE

G

The learning of students - development of students as learners in Yr 7 0.44 0.33 0.24 0.80 0.17 0.03

ACADEMY 1 Enfield 0.36 0.33 0.31 0.76 0.19 0.06

ACADEMY 2 Lords Hill 0.26 0.37 0.36 0.63 0.27 0.10

ACADEMY 3 Wintringham 0.44 0.33 0.24 0.80 0.17 0.03

GENDER 1 Male – not included

GENDER 2 Female 0.44 0.33 0.24 0.80 0.17 0.03

S1 1 Changing & Learning 0.82 0.16 0.03 0.85 0.14 0.01

S1 2 Learning Relationships 0.57 0.32 0.10 0.66 0.29 0.05

S1 3 Strategic Awareness 0.60 0.27 0.13 0.71 0.25 0.04

S1 4 Resilience 0.19 0.31 0.50 0.12 0.26 0.62

S1 5 Creativity 0.39 0.42 0.19 0.82 0.15 0.03

S1 6 Meaning Making 0.47 0.42 0.11 0.83 0.16 0.02

S1 7 Critical Curiosity 0.44 0.33 0.24 0.80 0.17 0.03

TABLE 147 STRAND 2 FULL RESULTS

196

Male students

Quest Statement PO

S

UN

C

NEG

The learning of students 0.86 0.13 0.01

ACADEMY 1 Enfield 0.72 0.22 0.07

ACADEMY 2 Lords Hill 0.00 1.00 0.00

ACADEMY 3 Wintringham 0.86 0.13 0.01

GENDER 1 Male 0.86 0.13 0.01

S2 1 I feel included in the activities that are available to me in the Academy 0.53 0.33 0.15

S2 2 I feel safe in the Academy 0.40 0.60 0.00

S2 3 I feel valued and cared for as an individual in the Academy 0.79 0.18 0.04

S2 4 I like learning new things 0.79 0.18 0.04

S2 5 I usually concentrate on my learning 0.79 0.18 0.04

S2 6 I am hard to distract 0.52 0.33 0.15

S2 7 I like working with other students to help my learning 0.87 0.12 0.01

S2 8 I don’t distract other students 0.38 0.34 0.28

S2 9 I am confident in my learning 0.99 0.01 0.00

S2 10 I take a lot of care with my work 0.87 0.12 0.01

S2 11 I work hard 0.79 0.18 0.04

S2 12 I do more than I am asked to do in class 0.78 0.18 0.04

S2 13 I don’t mind making mistakes because I learn from them 0.96 0.04 0.00

S2 14 I come up with new ideas to help my learning 0.50 0.33 0.17

S2 15 I enjoy my learning 0.78 0.18 0.04

S2 16 My teachers teach well so that I learn successfully 0.96 0.04 0.00

S2 17 I have regular opportunities to express my opinion to my teachers about my learning 0.96 0.04 0.00

S2 18 I often have conversations with my teachers which help me to make better progress 0.96 0.04 0.00

S2 19 My teachers seem to enjoy teaching my classes 0.78 0.18 0.04

S2 20 My teachers often share things that they have learned themselves 0.77 0.18 0.04

S2 21 I get on well with my teachers 0.96 0.04 0.00

S2 22 I get extra support to help my learning when I need it 0.50 0.33 0.17

S2 23 I know that the senior teachers want us to learn successfully 0.79 0.18 0.04

S2 24 Everyone works very hard to make the Academy a great place to learn and to do well 0.52 0.33 0.15

S2 25 My teachers constantly expect me to improve on my personal best 0.96 0.04 0.00

S2 26 Right now I am achieving the best I possibly can 0.90 0.10 0.00

S2 27 From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the Academy

0.87 0.12 0.01

S2 28 I have kept up a good rate of progress ever since my first year at the Academy 0.78 0.18 0.04

S2 29 I think that my results at the moment are as good as I can do 0.53 0.33 0.15

S2 30 I feel that my successes at the Academy are recognised 0.51 0.33 0.16

S2 31 I feel that my successes with activities outside the Academy are recognised 0.91 0.08 0.01

S2 32 I feel proud about what I have achieved so far 0.96 0.04 0.00

S2 33 We are taught about the responsibilities of students as well as their rights 0.78 0.18 0.04

S2 34 We care about helping each other in the Academy community 0.78 0.18 0.04

S2 35 We are encouraged to care about the needs of other people in the local community and around the world

0.79 0.18 0.04

S2 36 I regularly offer to help others 0.59 0.37 0.04

S2 37 My parents or carers feel that they are always welcome in the Academy 0.87 0.12 0.01

S2 38 My parents or carers feel involved with my learning and my progress 0.96 0.04 0.00

S2 39 My parents or carers are pleased with my progress 0.79 0.18 0.04

S2 40 My parents or carers have always been able to sort out any problems that I have had 0.52 0.33 0.15

S2 41 I am encouraged to lead activities in the classroom 0.55 0.35 0.10

S2 42 I have had opportunities to lead other activities at the Academy 0.86 0.13 0.01

Female students

Q Statement PO

S

UN

C

NEG

The learning of students 0.39 0.55 0.06

ACADEMY 1 Enfield 0.84 0.14 0.02

ACADEMY 2 Lords Hill 0.00 1.00 0.00

ACADEMY 3 Wintringham 0.39 0.55 0.06

GENDER 2 Female 0.39 0.55 0.06

S2 1 I feel included in the activities that are available to me in the Academy 0.58 0.29 0.13

S2 2 I feel safe in the Academy 0.99 0.01 0.00

197

S2 3 I feel valued and cared for as an individual in the Academy 0.25 0.33 0.42

S2 4 I like learning new things 1.00 0.00 0.00

S2 5 I usually concentrate on my learning 0.87 0.12 0.01

S2 6 I am hard to distract 0.16 0.32 0.52

S2 7 I like working with other students to help my learning 0.36 0.56 0.07

S2 8 I don’t distract other students 0.48 0.35 0.17

S2 9 I am confident in my learning 0.99 0.01 0.00

S2 10 I take a lot of care with my work 0.90 0.10 0.00

S2 11 I work hard 0.88 0.12 0.01

S2 12 I do more than I am asked to do in class 0.63 0.26 0.11

S2 13 I don’t mind making mistakes because I learn from them 0.63 0.26 0.11

S2 14 I come up with new ideas to help my learning 0.27 0.47 0.26

S2 15 I enjoy my learning 0.82 0.15 0.03

S2 16 My teachers teach well so that I learn successfully 0.70 0.22 0.08

S2 17 I have regular opportunities to express my opinion to my teachers about my learning 0.47 0.34 0.20

S2 18 I often have conversations with my teachers which help me to make better progress 0.17 0.35 0.48

S2 19 My teachers seem to enjoy teaching my classes 0.46 0.36 0.18

S2 20 My teachers often share things that they have learned themselves 0.91 0.09 0.01

S2 21 I get on well with my teachers 0.79 0.17 0.04

S2 22 I get extra support to help my learning when I need it 0.08 0.22 0.70

S2 23 I know that the senior teachers want us to learn successfully 0.97 0.03 0.00

S2 24 Everyone works very hard to make the Academy a great place to learn and to do well 0.74 0.20 0.06

S2 25 My teachers constantly expect me to improve on my personal best 0.94 0.05 0.00

S2 26 Right now I am achieving the best I possibly can 0.88 0.12 0.01

S2 27 From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the Academy

0.84 0.14 0.02

S2 28 I have kept up a good rate of progress ever since my first year at the Academy 0.81 0.15 0.03

S2 29 I think that my results at the moment are as good as I can do 0.83 0.15 0.03

S2 30 I feel that my successes at the Academy are recognised 0.66 0.24 0.09

S2 31 I feel that my successes with activities outside the Academy are recognised 0.34 0.60 0.06

S2 32 I feel proud about what I have achieved so far 0.89 0.11 0.01

S2 33 We are taught about the responsibilities of students as well as their rights 0.93 0.07 0.01

S2 34 We care about helping each other in the Academy community 0.88 0.11 0.01

S2 35 We are encouraged to care about the needs of other people in the local community and around the world

1.00 0.00 0.00

S2 36 I regularly offer to help others 0.82 0.15 0.03

S2 37 My parents or carers feel that they are always welcome in the Academy 0.95 0.05 0.00

S2 38 My parents or carers feel involved with my learning and my progress 0.73 0.20 0.07

S2 39 My parents or carers are pleased with my progress 0.89 0.11 0.00

S2 40 My parents or carers have always been able to sort out any problems that I have had 0.98 0.02 0.00

S2 41 I am encouraged to lead activities in the classroom 0.45 0.36 0.19

S2 42 I have had opportunities to lead other activities at the Academy 0.39 0.55 0.06

TABLE 148 STRAND 4 FULL RESULTS

Quest Statement POS UNC NEG

S4 Learning of students post 16 transition and progress to adulthood 0.85 0.14 0.02

ACADEMY 1 Enfield 0.85 0.14 0.02

ACADEMY 2 Lords Hill 0.00 1.00 0.00

ACADEMY 3 Wintringham 0.94 0.05 0.00

GENDER 1 Male 0.89 0.11 0.00

198

GENDER 2 Female 0.36 0.56 0.09

S4 1 1. I was valued and included during my time at the Academy 0.89 0.11 0.00

S4 2 2. I felt part of a community that was focused on learning 0.89 0.11 0.00

S4 3 3. I felt safe at the Academy 0.89 0.11 0.00

S4 4 4. I enjoyed my time at the Academy 0.83 0.15 0.03

S4 5 5. I did as well as I had hoped in my GCSEs 0.88 0.11 0.01

S4 6 6. My teachers accurately predicted my GCSE results 0.86 0.13 0.01

S4 7 7. I am looking forward to the move to Sixth Form, Further Education,

training or work

0.50 0.33 0.18

S4 8 8. I have been well prepared for the next stage of my life 0.87 0.12 0.01

S4 9 9. My successes were recognised during my time at the Academy 0.89 0.11 0.00

S4 10 10. I was helped and supported whenever I struggled with my learning 0.98 0.02 0.00

S4 11 11. I developed a sense of responsibility towards other people during

my time at the Academy

0.89 0.11 0.00

S4 12 12. The Academy helped to make me more understanding about

different cultures and types of people

0.87 0.12 0.01

S4 13 13. I supported activities that would improve things for other people 0.61 0.38 0.01

S4 14 14. I was offered opportunities for leadership during my time at the

Academy

0.85 0.14 0.02

S4 15 15. I led activities during my time at the Academy 0.49 0.34 0.17

S4 16 16. My teachers taught me well at the Academy 0.90 0.10 0.00

S4 17 17. My teachers were always trying to improve their teaching 0.87 0.12 0.01

199

S4 18 18. I was regularly challenged to do better so that I could reach my full

potential

0.89 0.11 0.00

S4 19 19. My parents/carers felt involved with my education at the Academy 0.83 0.15 0.03

S4 20 20. My parents/carers felt that they could find the right person to talk

to at the Academy if they had any concerns about my education

0.87 0.12 0.01

S4 21 21. Senior staff at the Academy wanted us to learn well and achieve

our best

0.87 0.12 0.01

S4 22 22. Everyone at the Academy worked very hard to make it a great place

to learn and to do well

0.83 0.15 0.03

S4 23 23. The Academy was well run 0.87 0.12 0.01

S4 24 24. There were great facilities for learning 0.89 0.11 0.00

S4 25 25. There was a broad curriculum that helped me to develop as a whole

person

0.89 0.11 0.01

S4 26 26. I enjoyed a wide range of opportunities for learning beyond the

timetabled curriculum

0.89 0.11 0.00

S4 27 27. The Academy made good use of new technologies for learning 0.90 0.10 0.00

S4 28 28. I never felt that anyone was putting limits on what I could achieve 0.85 0.13 0.02

S4 29 29. I feel proud about what I achieved at the Academy 0.87 0.12 0.01

S4 30 30. I could not have gone to a better place for my secondary education 0.85 0.14 0.02

TABLE 149 STRAND 5 FULL RESULTS

Quest Statement POS UNC NEG

Parents/ Carers 2013 (STRAND 5) 0.56 0.36 0.08

200

Acad1 2013 0.56 0.36 0.08

Acad2 2013 0.00 1.00 0.00

Acad3 2013 0.00 1.00 0.00

Gen1 2013 0.00 1.00 0.00

Gen2 2013 0.00 1.00 0.00

S5 1 1. My child is known and valued as an individual 0.87 0.12 0.01

S5 2 2. The Academy is providing for my child’s particular needs 0.37 0.57 0.06

S5 3 3. My child feels included in the life of the Academy, in classrooms and

in other activities

0.35 0.55 0.10

S5 4 4. The Academy focuses its work on providing the best opportunities

for learning for all students

0.37 0.57 0.06

S5 5 5. My child is taught well 0.59 0.37 0.04

S5 6 6. My child is being challenged to improve on previous best

performance in all areas

0.59 0.37 0.04

S5 7 7. My child’s progress at the Academy has exceeded our expectations 0.43 0.30 0.27

S5 8 8. My child’s successes are recognised 0.59 0.37 0.04

S5 9 9. The Academy ensures my child is well looked after 0.59 0.37 0.04

S5 10 10. The Academy helps my child to have a healthy lifestyle 0.37 0.57 0.07

S5 11 11. There is a good standard of behaviour at the Academy 0.50 0.34 0.15

S5 12 12. My child is developing an attitude of helping others (for example,

by doing practical activities to improve life at the Academy, by

supporting people in need, getting involved with community projects,

fund-raising)

0.58 0.36 0.06

201

S5 13 13. My child is taking opportunities to lead activities (for example,

leading group work in the classroom, leading a sports team, giving

feedback to teachers about their learning, being a representative on a

Student Council, taking a lead role in Academy performances of music,

drama or dance, being a coach, mentor or ‘buddy’)

0.24 0.43 0.33

S5 14 14. My child has kept up a good rate of progress at key transitions (for

example, from primary to secondary school; from Year 9 to GCSE

courses in Years 10 & 11; from Year 11 to Sixth Form/College; from Year

13 to further education/higher education/employment)

0.35 0.55 0.10

S5 15 15. I get regular and helpful information about my child’s progress 0.59 0.37 0.04

S5 16 16. The Academy responds well to my concerns 0.59 0.36 0.05

S5 17 17. I always find staff knowledgeable about my child 0.59 0.36 0.05

S5 18 18. I always find staff helpful when discussing my child’s progress 0.59 0.36 0.05

S5 19 19. The Academy helps me to support my child’s learning 0.59 0.37 0.04

S5 20 20. The Academy is led and managed effectively 0.59 0.37 0.04

S5 21 21. My impression is that senior staff focus their work on making the

Academy a good place to learn

0.59 0.37 0.04

S5 22 22. My impression is that senior staff expect all students to achieve

well

0.59 0.37 0.04

S5 23 23. My child enjoys their education at the Academy 0.39 0.59 0.02

S5 24 24. I would recommend the Academy to another parent/carer 0.56 0.36 0.08

S5 25 (25)26. The Academy has been helpful in ensuring that I understand the

general information sent to parents/carers

0.00 1.00 0.00

S5 26 (26)27. The Academy has been helpful in ensuring that I understand the

information about my child’s progress

0.00 1.00 0.00

S5 27 (27)28. I feel that the Academy understands and takes proper account 0.00 1.00 0.00

202

of the particular issues and sensitivities of my culture

S5 28 (28)29. If I have a concern, I feel confident that I can make contact with

relevant staff

0.00 1.00 0.00

S5 29 (29)30. If I have a concern, I feel confident that I can communicate with

staff effectively

0.00 1.00 0.00

S5 30 (30)31. I would recommend the Academy to another parent/carer from

my community

0.00 1.00 0.00

TABLE 150 STRAND 6 FULL RESULTS

Quest Statement POS UNC NEG

ECHO Project Strand 6 – The Learning of Teachers 0.90 0.10 0.00

ACADEMY 1 Enfield 0.90 0.10 0.00

ACADEMY 2 Lords Hill 0.00 1.00 0.00

ACADEMY 3 Wintringham 0.00 1.00 0.00

GENDER 1 Male 0.94 0.05 0.00

GENDER 2 Female 0.89 0.11 0.00

S6 A1 Staff as well as students learn in this academy 0.92 0.07 0.01

S6 A2 I draw on good practice from other schools as a means to further my

own professional development

0.34 0.43 0.24

S6 A3 I read research reports as one source of useful ideas for improving my

practice

0.31 0.32 0.37

S6 A4 I use the web as one source of useful ideas for improving my practice 0.77 0.19 0.05

S6 A5 I consult my students about how they learn most effectively 0.59 0.37 0.04

203

S6 A6 I relate what works in my own practice to research findings 0.78 0.18 0.04

S6 A7 I am able to see how practices that work in one context might be

adapted to other contexts

0.85 0.13 0.02

S6 A8 I use insights from my own professional learning to feed into academy

policy development

0.52 0.33 0.14

S6 A9 I reflect on my practice as a way of identifying my professional

learning needs

0.86 0.13 0.02

S6 A10 I experiment with my practice as a conscious strategy for improving

classroom teaching and learning

1.00 0.00 0.00

S6 A11 I modify my practice in the light of feedback from my students 0.89 0.11 0.00

S6 A12 I modify my practice in the light of published research evidence 0.52 0.33 0.16

S6 A13 I modify my practice in the light of evidence from self-evaluations of

my classroom practice

0.86 0.13 0.02

S6 A14 I modify my practice in the light of evidence from evaluations of my

classroom practice by managers or other colleagues

0.87 0.12 0.01

S6 A15 I carry out joint research/evaluation with one or more colleagues as a

way of improving my practice

0.33 0.31 0.36

S6 A16 I collaborate to plan my teaching 0.83 0.15 0.03

S6 A17 I observe other colleagues in the classroom and we give each other

feedback

0.58 0.36 0.06

S6 A18 I engage in team teaching as a way of improving practice 0.29 0.49 0.22

S6 A19 If I have a problem with my teaching I turn to colleagues for help 0.85 0.14 0.02

S6 A20 I suggest ideas or approaches for colleagues to try in class 0.35 0.55 0.10

S6 A21 I make collective agreements to test out new ideas 0.51 0.33 0.16

204

S6 A22 I discuss openly with colleagues what and how they are learning 0.59 0.36 0.05

S6 A23 I use informal opportunities to discuss how children learn 0.51 0.33 0.16

S6 A24 I offer reassurance and support to colleagues 0.79 0.18 0.04

S6 A25 My practice is based on the belief that all students are capable of

learning

1.00 0.00 0.00

S6 A26 My experience is that students in this academy enjoy learning 0.77 0.18 0.05

S6 A27 My experience is that student success is celebrated 0.87 0.12 0.01

S6 A28 I discuss with colleagues how students might be helped to learn how

to learn

0.62 0.38 0.01

S6 B1 Senior leaders communicate a clear vision of where the academy is

going

0.99 0.01 0.00

S6 B2 Staff have a commitment to the whole academy as well as to their

department, key stage and/or year group

0.92 0.08 0.00

S6 B3 Senior leaders promote commitment among staff to the whole

academy as well as to the department, key stage and/or year group

0.19 0.37 0.44

S6 B4 There is effective communication between senior leaders and

teachers

0.58 0.29 0.13

S6 B5 There are processes for involving all staff in decision-making 0.57 0.29 0.14

S6 B6 Teachers’ professional know-how is used in the formulation of

academy policy and goals

0.87 0.12 0.01

S6 B7 Teachers’ professional know-how is used in the formulation of

academy policy, even where this leads to a questioning of established

rules, procedures and practices

0.52 0.32 0.16

S6 B8 Opportunities are provided for teachers to critically evaluate academy

policy

0.64 0.33 0.03

205

S6 B9 Staff are involved in evaluating academy policy 0.32 0.43 0.25

S6 B10 Staff participate in important decision-making 0.34 0.33 0.32

S6 B11 There are processes for involving students in decision-making 0.56 0.30 0.14

S6 B12 Staff have a good working knowledge of the Academy Development

Plan (or your academy’s equivalent document)

0.85 0.13 0.02

S6 B13 Staff see the Academy Development Plan as relevant and useful to

learning and teaching

0.53 0.35 0.12

S6 B14 Staff development time is used effectively to realise Academy

Development Plan priorities

0.62 0.27 0.12

S6 B15 Staff development time is used effectively in the academy 0.04 0.16 0.81

S6 B16 The academy provides cover to allow staff joint planning time 0.90 0.10 0.00

S6 B17 Teachers are encouraged to experiment with new ideas as a way of

promoting professional growth

0.84 0.14 0.02

S6 B18 Formal training provides opportunities for teachers to develop

professionally

0.85 0.13 0.02

S6 B19 Teachers are helped to develop skills to assess students’ work in ways

that move students on in their learning

0.52 0.35 0.13

S6 B20 Teachers are helped to develop skills to observe learning as it happens

in the classroom

0.44 0.37 0.19

S6 B21 Senior leaders support teachers in sharing practice with other schools

through networking

0.53 0.32 0.16

S6 B22 Information is collected from teachers on those aspects of their work

that they themselves think they do effectively

0.87 0.12 0.01

S6 B23 Information is collected from teachers on effective ways they promote

learning to learn skills and knowledge among their students

0.14 0.35 0.51

206

S6 B24 Information is collected from teachers on informal teacher networking

in which they play an active role

0.22 0.33 0.45

S6 B25 Teacher-initiated networking is an integral element of staff

development

0.22 0.40 0.38

S6 C1 I feel included in a community that values what I can offer to the staff

team and to students

0.99 0.01 0.00

S6 C2 I believe that I am working in a community that focuses our attention

and activities on learning to improve teaching and student outcomes

0.95 0.05 0.00

S6 C3 My experience suggests that other members of staff are involved in

training and development activities to improve their classroom

practice and student outcomes

0.65 0.25 0.10

S6 C4 I have had opportunities to work alongside and/or train with

colleagues in dother phases to improve students’ progress

0.99 0.01 0.00

S6 C5 My experience is that senior leaders are role models as ‘leaders of

learning’ in the academy

0.98 0.02 0.00

S6 C6 Leadership is shared in this academy and opportunities to take on

leadership roles are available to all those that want them and have the

relevant knowledge and experience

0.81 0.16 0.04

S6 C7 My experience is that the academy promotes a balance between the

rights and the responsibilities of all members of the community

0.72 0.21 0.07

S6 C8 There is an emphasis among students and staff on caring about and

helping other people in the academy community

0.90 0.09 0.01

S6 C9 There is an emphasis among students and staff on caring about and

helping other people in the wider community

0.91 0.09 0.01

S6 C10 The academy is successful in helping parents/carers to support the

learning of their children

0.40 0.33 0.26

S6 C11 The achievements of staff are recognised and celebrated 0.52 0.35 0.14

S6 C12 I believe that I am realising my full potential as a teacher 0.90 0.10 0.00

207

TABLE 151 STRAND 7 FULL RESULTS

Quest Statement POS UNC NEG

ECHO Strand 7: Leadership of learning 0.90 0.10 0.00

ACADEMY 1 Enfield 0.98 0.02 0.00

ACADEMY 2 Lords Hill 0.90 0.10 0.00

ACADEMY 3 Wintringham 0.83 0.14 0.03

GENDER 1 Male 0.00 1.00 0.00

GENDER 2 Female 0.00 1.00 0.00

S7 1 1. I have a good understanding of my own and other people’s

learning

1.00 0.00 0.00

S7 2 2. I keep my knowledge about teaching and learning and about my

phase and/or subject specialism up to date

1.00 0.01 0.00

S7 3 3. I teach regularly and my lessons are rated good or outstanding 0.81 0.18 0.01

S7 4 4. I undertake regular and substantial professional development for

myself

0.89 0.11 0.01

S7 5 5. I understand the differences between good and outstanding

lessons

1.00 0.00 0.00

S7 6 6. I regularly work with staff to improve their teaching practice 0.77 0.18 0.05

S7 7 7. I can design, monitor and assess programmes for effective

teaching and learning

0.89 0.11 0.01

S7 8 8. I regularly participate in critical and reflective enquiry to develop

teaching and learning across the academy

0.37 0.57 0.07

S7 9 9. I understand what an academy needs to do to become good and 0.90 0.10 0.00

208

outstanding

S7 10 10. I regularly work with staff to help them improve their leadership

and management

0.89 0.11 0.00

S7 11 11. I am an effective coach and use my coaching skills regularly to

help others to improve their performance

0.89 0.10 0.00

S7 12 12. I regularly interact with staff in other schools, academies and/or

organisations to build learning partnerships which connect staff and

students to learning opportunities beyond our academy

0.29 0.48 0.24

S7 13 13. I make regular and effective contributions to educational dialogue

that shapes whole academy policy and informs practice

0.87 0.12 0.01

S7 14 14. I contribute effectively to the critical development of academy

systems and structures in order to ensure quality teaching focused on

student progress in learning

0.89 0.11 0.00

S7 15 15. The senior leadership team communicates a clear vision of where

the academy is going so that there is a shared sense of purpose

0.90 0.10 0.00

S7 16 16. Staff have a commitment to the whole academy as well as to

their department, key stage and/or year group

0.89 0.11 0.00

S7 17 17. The senior leadership team effectively promotes commitment

among staff to the future development of the whole academy

0.90 0.10 0.00

S7 18 18. There is effective communication between senior leaders and

teachers

0.61 0.38 0.01

S7 19 19. There are processes for involving all staff in decision-making 0.61 0.38 0.01

S7 20 20. Teachers’ professional know-how is used in the formulation of

academy policy and goals

0.61 0.38 0.02

S7 21 21. Teachers’ professional know-how is used in the formulation of

academy policy, even where this leads to a questioning of established

rules, procedures and practices

0.57 0.36 0.07

S7 22 22. Opportunities are provided for teachers to critically evaluate 0.58 0.36 0.06

209

academy policy

S7 23 23. Staff participate in important decision-making 0.61 0.38 0.02

S7 24 24. There are processes for involving students in decision-making,

including how best to improve classroom practice

0.90 0.10 0.00

S7 25 25. Staff have a good working knowledge of the Academy

Development Plan

0.61 0.38 0.01

S7 26 26. Staff see the Academy Development Plan as relevant and

useful to learning and teaching

0.61 0.38 0.02

S7 27 27. Staff development time is used effectively to realise Academy

Development Plan priorities

0.73 0.20 0.06

S7 28 28. Staff development time is used effectively in the academy,

particularly through the explicit modelling of good practice

0.59 0.36 0.05

S7 29 29. The academy provides support to allow staff joint planning

time

0.57 0.36 0.07

S7 30 30. Teachers are encouraged to experiment with new ideas as a

way of promoting professional growth and improving classroom

practice

0.85 0.13 0.02

S7 31 31. Formal training provides opportunities for teachers to develop

professionally

0.86 0.13 0.02

S7 32 32. Teachers are helped to develop skills to assess students’ work

in ways that move their students on in their learning

0.58 0.36 0.06

S7 33 33. Teachers are helped to develop skills to observe learning as it

happens in the classroom

0.57 0.36 0.07

S7 34 34. Senior leaders support teachers in sharing practice with other

schools and academies through networking

0.57 0.36 0.07

S7 35 35. Information is collected from teachers on those aspects of

their work that they themselves think they do effectively

0.87 0.12 0.01

210

S7 36 36. Information is collected from teachers on effective ways they

promote ‘learning to learn’ skills and knowledge among their students

0.80 0.16 0.04

S7 37 37. Information is collected from teachers on informal teacher

networking in which they play an active role

0.79 0.17 0.04

S7 38 38. Teacher-initiated networking is an integral element of staff

development

0.79 0.17 0.04

S7 39 39. ‘Learning how to learn’ is an issue discussed in staff

development time

0.52 0.35 0.13

S7 40 40. There is a well-developed shared language and a rich dialogue

about teaching and learning across the academy

0.89 0.11 0.00

S7 41 41. We are good at systematic monitoring, use of data and

feedback across the academy

0.59 0.36 0.05

S7 42 42. We have efficient and effective support systems and

structures across the academy

0.90 0.10 0.00

S7 43 43. The senior leadership team are effective role models as

‘leaders of learning’, (for example, by securing broad commitment to

learning for all that is focused on improving the quality of teaching

and student outcomes)

0.58 0.36 0.06

S7 44 44. We have developed successful approaches to distributed

leadership across the academy where the pathways of accountability

are working well

0.86 0.13 0.02

S7 45 45. The academy is successful in helping parents/carers to support

the learning of their children

0.57 0.36 0.08

S7 46 46. The governors play a full and appropriate part in strategic

planning to improve teaching and learning and student outcomes

0.57 0.36 0.07

S7 47 47. The academy involves a wide range of other stakeholders

directly and powerfully in the education of students

0.61 0.38 0.01

S7 48 48. The academy is good at securing the broadest possible impact

of successful innovations in teaching and learning

0.57 0.36 0.07

211

S7 49 49. There is a widespread and well-established climate of trust in

the academy

0.85 0.13 0.02

S7 50 50. There is an appetite to face weaknesses and bring about

changes for improvement in the academy

0.85 0.13 0.02

S7 51 51. Good personal relationships and strong teams help us to face

challenges with confidence

0.98 0.02 0.00

S7 52 52. The senior leadership team has the skills and resilience to

successfully work through the more complex challenges that we are

facing

0.90 0.10 0.00

S7 53 53. The senior leadership team is good at long term planning for

capacity building (for example, successfully addressing issues like

student progress across key transitions; improving the ‘tail’ of student

underachievement and/or significant achievement gaps; succession

planning)

0.90 0.10 0.00

S7 54 54. As an academy, we are good at celebrating success 0.90 0.10 0.00

APPENDIX 6

EXTRACT FROM DAVIS AND FLETCHER (2000)

The method [of propagation of evidence] is based on the Interval Probability Theory of Cui & Blockley (1990). This itself

is a development of earlier ideas of using intervals by Dempster (1969) and Shafer (1976).

Consider the conjecture, H, that a process is going to be successful. In the ideal situation the dependability of a process

to deliver its objectives is certain. However in reality the certainty depends on the dependability of its sub-processes

about which we need evidence. Suppose then also that there is evidence from a sub-process E. We are concerned with

the probability P(H) that H will be successful. The situation is illustrated using the Venn diagrams in Figure A.

Interestingly, the last condition (c) is the most general and can be considered as a generalisation of (b) since the

evidence is now only partially sufficient. The computation of the contribution of the evidence E to the conjecture H has

been implemented in the method of Hall et al (1998), considering only the accumulation of evidence from the sub-

processes adding to the knowledge about H.

P(H) = P(H | E).P(E)

It is necessary to look at what happens when two or more pieces of evidence from two or more sub-processes are

accumulated. It can be seen from Figure B that there are a number of variables that need to be known to deduce the

212

probability of the success of the process (conjecture H). Firstly, the probability of the evidence itself being dependable

is required. This may be elicited directly or it may come from its own daughter processes. This is P(E). Then the

relevance or level of ‘sufficiency’ of E1 and E2 to H, or P(H | E)is needed.

When there is more than one sub-process, the dependency or overlap between the two is needed. This is so that

common evidence is not counted twice. In the case of the total probability method of Hall et al (1998), further

judgements are required to specify the P(H | E)terms.

The necessary condition expressed in Figure A comes out naturally as part of the total probability approach in that if for

example both E1 and E2 are necessary conditions, then:

where the value [0,0] represents the interval probability with meaning no support for the conjecture and full support

against, i.e. since E1 and E2 are necessary, anything not E1 and not E2 has no contribution to make.

The method implemented in the software has no way of expressing this, and so a heuristic has been introduced to

express the weight of the necessary condition (Figure A) as no P(H | E)terms are used. The significance of the

heuristic in practice is to recognise strong evidence against the success of a process that may lead ultimately to the

failure of the parent process.

The algorithms used in the software uses the heuristic described above and only single probability numbers for the

expressions of ‘sufficiency’ and ‘dependency ‘and a Boolean operator for necessity. The advantages [over other

algorithms] are that process models can be built and explored very quickly and have been found in practice to be

powerful tools for illuminating the decision making process.

Figure A: Venn diagrams of the logical conditions (a) E necessary for H, (b) E sufficient for E, and (c) E relevant to H respectively.

Figure B: The contributions of two sub-processes E1 & E2 to the conjecture H.