USING EDUCATIONAL EFFECTIVENESS RESEARCH TO IMPROVE THE QUALITY OF EDUCATION: THE CONTRIBUTION OF THE DYNAMIC MODEL Leonidas Kyriakides Leonidas Kyriakides

USING EDUCATIONAL EFFECTIVENESS USING EDUCATIONAL EFFECTIVENESS RESEARCH TO IMPROVE THE QUALITY RESEARCH TO IMPROVE THE QUALITY OF EDUCATION: THE CONTRIBUTION OF EDUCATION: THE CONTRIBUTION

OF THE DYNAMIC MODELOF THE DYNAMIC MODEL

Leonidas KyriakidesLeonidas Kyriakides

Department of Education, Department of Education,

University of CyprusUniversity of Cyprus

INTRODUCTION: HISTORY OF INTRODUCTION: HISTORY OF EDUCATIONAL EFFECTIVENESS RESEARCHEDUCATIONAL EFFECTIVENESS RESEARCH

Educational Effectiveness Research (EER) addresses the questions on what works in education and why.

Over the last two decades EER has been improved considerably by the criticism on research design, the sampling and statistical techniques.

Methodological advances have enabled more efficient estimates of teacher and school differences in student achievement to be obtained (Goldstein, 2003).

Progress was made by a more precise definition of the concepts used and the relations between the concepts (Scheerens & Bosker, 1997).

INTRODUCTION: HISTORY OF INTRODUCTION: HISTORY OF EDUCATIONAL EFFECTIVENESS RESEARCHEDUCATIONAL EFFECTIVENESS RESEARCH

The whole process has not contributed significantly to the improvement of school effectiveness.

The dynamic model of educational effectiveness attempts to define the dynamic relations between the multiple factors found to be associated with effectiveness.

The establishment of the dynamic model and its empirical testing is expected to help EER to establish stronger links with school improvement (Creemers & Kyriakides, 2006).

PRESENTATION OUTLINEPRESENTATION OUTLINE

1) The dynamic model of educational effectiveness: an overview

2) Testing the validity of the dynamic model: Findings and new research questions

3) Using the dynamic model to develop an Evidence-Based and Theory-Driven Approach to school improvement

4) Using the Dynamic Model for Improvement Purposes: Studies currently being undertaken

THE DYNAMIC MODEL OF EDUCATIONAL THE DYNAMIC MODEL OF EDUCATIONAL EFFECTIVENESS: AN OVERVIEWEFFECTIVENESS: AN OVERVIEW

Teaching and learning are dynamic processes that are constantly adapting to changing needs and opportunities. Effective schooling should be treated as a dynamic, ongoing process.

The model is multilevel in nature and refers to multiple factors of effectiveness which operate at four levels.

The teaching and learning situation is emphasized.

Essential characteristics of the model


School-level factors are expected to influence the teaching-learning situation by developing and evaluating the school policy on teaching and the policy on creating a learning environment at the school.

The final level refers to the influence of the educational system through a more formal way, especially through developing and evaluating the educational policy at the national/regional level.



Factors at the school and context level have both direct and indirect effects on student achievement.

Some factors which operate at the same level are related to each other.

Each factor is defined and measured by taking Each factor is defined and measured by taking into account five dimensions: into account five dimensions: frequency, frequency, focus, stage, quality, focus, stage, quality, andand differentiation differentiation..


USING A MULTIDIMENSIONAL APPROACH TO USING A MULTIDIMENSIONAL APPROACH TO MEASURE THE FUNCTIONING OF FACTORSMEASURE THE FUNCTIONING OF FACTORS

FrequencyFrequency refers to the quantity that an activity associated with an effectiveness factor is present in a system, school or classroom. This dimension may not always be related in a linear way with student outcomes.

Personal monitoring at school level can be measured by taking into account how often the principles use a monitoring system to supervise their teachers.

The other four dimensions examine qualitative characteristics of the functioning of the factors and help us describe the complex nature of educational effectiveness.

USING A MULTIDIMENSIONAL APPROACH USING A MULTIDIMENSIONAL APPROACH TO MEASURE THE FUNCTIONING OF TO MEASURE THE FUNCTIONING OF

FACTORSFACTORSTwo aspects of Two aspects of focusfocus are seen as important. The first are seen as important. The first one refers to the one refers to the specificity specificity of the activities which can of the activities which can range from very specific range from very specific to general (e.g. a structuring task may either refer to a part of a lesson or to the whole lesson or even to a series of lessons).The second aspect of this dimension addresses the The second aspect of this dimension addresses the purpose purpose for which an activity takes place. An activity for which an activity takes place. An activity may be expected to achieve a may be expected to achieve a single or multiple single or multiple purposes.purposes.If all the activities are expected to achieve a single purpose, then the chances to achieve this purpose are high, but the effect of the factor might be small due to the fact that other purposes are not achieved and synergy may not exist. There should be a balance with respect to the two There should be a balance with respect to the two aspects of the focus dimension.aspects of the focus dimension.

USING A MULTIDIMENSIONAL APPROACH TO USING A MULTIDIMENSIONAL APPROACH TO MEASURE THE FUNCTIONING OF FACTORSMEASURE THE FUNCTIONING OF FACTORS

The stagestage at which tasks associated with a factor take place is examined. Factors need to take place over a long period of time to ensure that they have a continuous direct or indirect effect on student learning.

School policy on opportunity to learn should be implemented throughout the year and not only through specific regulations announced at a specific point of time.

Measuring the stage dimension gives information about the continuity of the existence of a factor but the activities associated with the factor may not necessarily be the same.The qualityquality refers to properties of the specific factor itself, as these are discussed in the literature. DifferentiationDifferentiation refers to the extent to which activities associated with a factor are implemented in the same way for all the subjects involved with it. Adaptation to specific needs of each subject or group of subjects will increase the successful implementation of a factor and maximize its effect on learning.

TEACHERTEACHER FACTORS OF THE DYNAMIC FACTORS OF THE DYNAMIC MODELMODEL

Teacher factors refer to teachers’ instructional role and were found to be associated with student outcomes (e.g., Brophy & Good, 1986; Muijs & Reynolds, 2001; Seidel & Shavelson, 2007). The eight classroom factors are as follows: orientation, structuring, questioning, teaching-modelling, applications, management of time, teacher role in making classroom a learning environment, and classroom assessment. The eight teacher factors do not refer only to one approach of teaching such as the direct and active teaching approach or the constructivist approach but cover at least partly the main approaches to learning and teaching.

SCHOOLSCHOOL FACTORS OF THE DYNAMIC MODEL FACTORS OF THE DYNAMIC MODEL

Emphasis is given to two main aspects of the school policy which affect learning at both the level of teachers and students: a) school policy for teaching and b) school policy for creating a learning environment at school. The factors concerned with the school policy mainly refer to the actions taken by the school to help teachers and other stakeholders have a clear understanding of what is expected from them to do. Support offered to teachers and other stakeholders to implement the school policy is also an aspect of these two overarching factors.

SCHOOLSCHOOL FACTORS OF THE DYNAMIC MODEL FACTORS OF THE DYNAMIC MODEL

The dynamic model refers to the following four overarching school factors:

School policy for teaching and actions taken for improving teaching practice, Evaluation of school policy for teaching and of actions taken to improve teaching, Policy for creating a school learning environment and actions taken for improving the school learning environment, Evaluation of the school learning environment

TESTING THE VALIDITY OF THE MODEL: TESTING THE VALIDITY OF THE MODEL: FINDINGS AND NEW RESEARCH FINDINGS AND NEW RESEARCH

QUESTIONS QUESTIONS 1. A longitudinal studylongitudinal study testing the validity of the model was

conducted (Kyriakides & Creemers, 2008). Participants: All grade 5 students (n=2503) from each class

(n=108) of 50 primary schools in Cyprus.

Achievement in mathematics, language, and religious education were measured in September 2004, May 2005, and May 2006.

It was possible to provide evidence supporting the validity of the proposed measurement framework.

The importance of using five dimensionsfive dimensions to measure the teacher and school factors was also identified (Creemers & Kyriakides, 2010).

The impact of school factors depends on the current situationsituation of the school and on the problems that it is facing.

In schools where quality of teaching is rather low, school factors had stronger effects on student outcomes.


QUESTIONS QUESTIONS 2. A longitudinal study investigating the impact of teacher factors

on achievement of students at the end of pre-primary school was conducted (Kyriakides & Creemers, 2009). a. Similarities and differences in effective teaching of two

different subjects (mathematics and Greek language) and at two different phases of schooling (pre-primary and primary education) were identified.

b. The assumption that teacher factors are genericgeneric was mainly supported but some factors were found to be more important for one age of schooling than another.

3. A quantitative synthesis of school effectiveness studies conducted during the last 25 years provided some support to the validity of the model at the school levelschool level. Factors excluded from the model were only weakly associated with student achievement (Kyriakides, Creemers, Antoniou & Demetriou, 2010).


QUESTIONS QUESTIONS

4. We conducted a replication study in the same 50 primary schools where the original study investigating the validity of the dynamic model was conducted. The design of the study was identical to that of the original study.This study investigates one of the essential differences of the dynamic model which has to do with its attempt to relate changes in the effectiveness status of schools to the changes in the functioning of school factors.

TESTING THE VALIDITY OF THE MODEL: TESTING THE VALIDITY OF THE MODEL: FINDINGS AND NEW RESEARCH QUESTIONS FINDINGS AND NEW RESEARCH QUESTIONS

The results of DFA revealed that we can predict changes in the effectiveness status of schools by looking at changes in the teaching practice and changes in the functioning of school factors (Creemers & Kyriakides, in press). We did not predict changes that occur in schools which remained among the most effective. Schools cannot remain among the most effective unless improvement in the functioning of school factors is observed. Further studies to test the generalizability of these results are needed.Effectiveness studies should not simply try to understand variation on the effectiveness status of schools during one academic year but explain why specific changes in their effectiveness status take place.


QUESTIONSQUESTIONSTo test the generalisability of the findings of these studies by investigating

the impact of factors upon different learning outcomesthe impact of factors upon achievement of different age group of children

To identify which factors of the dynamic model are associated with learning outcomes irrespective of the context and which factors have differential effects and are therefore more relevant for policy making in specific socio-cultural contexts.

To measure and identify the impact of system factors

THE ESF PROJECT: MAIN AIMSTHE ESF PROJECT: MAIN AIMS

To investigate and explain differences between countries and schools within countries in the average and differential added value of education for different learning outcomes.to inform policy makers about effective practices at system, school, and classroom level contributing to the improvement of educational quality in terms of higher average achievement and better opportunities for disadvantaged students.to test the validity of the dynamic model in relation to diversity of student intake, processes, and prospective outcomes in order to improve the effectiveness of education based on scientific validated models. to elaborate on the system level factors of the dynamic model, explore their relationships with educational outcomes and with the school and classroom level factors, and draw implications for policy and research.

THE ESF PROJECT: RESEARCH DESIGN THE ESF PROJECT: RESEARCH DESIGN AND METHODSAND METHODS

In each country, a sample of at least 50 primary 50 primary schoolsschools is drawn. Tests in mathematics and mathematics and sciencescience to all grade 4 students are administered at the beginning and at the end of school year 2010-2011.

Student level factorsStudent level factors: We restrict ourselves to prior-knowledge, SES, ethnicity, prior-knowledge, SES, ethnicity, and and gendergender.

These factors explain the majority of variance at student level and can be used to search for differential effects differential effects of classroom and school factors.

All classroom and school level factorsclassroom and school level factors of the model will be measured.


System level factorsSystem level factors:: A description of the actors at different layers of the system level in each country will be conducted. Based on this analysis, we will determine which actors have to be addressed in each country in order to get full information about the system factors operating in each country. Data on system level factors will be generated through not only a content analysis of policy content analysis of policy documentsdocuments but also by interviewsinterviews to policy makers and other stakeholders and professionals.A questionnaire will be developed to measure the perceived impact of national policyperceived impact of national policy on schools and will be administered to teachers and head teachers of the school sample in each country.


We will measure the perceptions of policy makers, teachers, and school leaders about what constitutes the wider educational environmentwider educational environment and how this affects policy at national and school level. Questionnaires will be administered to policy makers, teachers, and head teachers investigating the functioning of factors associated with the wider educational environment in each country.

Pilot studiesPilot studies will be conducted in each country in order to test the face and the construct validity of each instrument.

Structural equation modellingStructural equation modelling techniques will be used to test the construct validity of the instruments.

THE ESF PROJECT: ANALYSIS OF DATATHE ESF PROJECT: ANALYSIS OF DATA

Within-countryWithin-country analyses will be conducted in order to identify the extent to which each classroom- and school-level factor is associated with achievement in each outcome.

Across-countriesAcross-countries analysis will help us identify generic and differential factors operating at different levels. We will search for differential impactdifferential impact of factors for different groups of students in order to provide suggestions on how education addresses the diversity in the society. The impact of factors in relation to equityequity (i.e., reducing the gap between achievement of different groups) will be examined.

THE ESF PROJECT: POTENTIAL IMPACTTHE ESF PROJECT: POTENTIAL IMPACT

In each country data associated with the design of the common project will be collected. In this way, multilevel analysis of the data across and within each country will reveal the importance of factors of the dynamic model. The individual projects address topics that are relevant to specific aspects of the theoretical framework of the common project and provide in depth information on the functioning of the dynamic model in different contexts. The main aim of the ESF project is not only to generate a theory about the dynamic perspective of effective education but also to come up with possibilities of establishing an evidence-based and theory-drivenevidence-based and theory-driven approach in policy making.

USING THE DYNAMIC MODEL TO DEVELOP AN USING THE DYNAMIC MODEL TO DEVELOP AN EVIDENCE-BASED AND THEORY-DRIVEN EVIDENCE-BASED AND THEORY-DRIVEN APPROACH TO SCHOOL IMPROVEMENTAPPROACH TO SCHOOL IMPROVEMENT

The dynamic model of educational effectiveness could contribute in establishing a theory-driven and evidence-based approach to school improvement.

The model does not only refer to factors that are important for explaining variation in student achievement but it also attempts to explain why these factors are important by integrating different theoretical orientations to effectiveness (Sammons, 2009).

Teachers are offered the opportunity to use in a flexible way this knowledge-base, adapt it to their specific needs, and develop their own strategies for school improvement (Teddlie & Stringfield, in press).

The model represents the complexity of educational effectiveness but this representation in factors and dimensions of factors provides opportunities to address improvement of education in a flexible way.

By treating differentiation as a dimension of measuring the functioning of each factor.

The model points at the possibility of defining grouping of factors. This implies that more comprehensive strategies with synergetic effects can be developed in order to address the improvement needs of each school.


During the last eight years, studies attempting to develop and test the dynamic model have been conducted and provided support to the validity of the dynamic model (Creemers & Kyriakides, in press).

Since the dynamic model was designed in order to establish stronger links between EER and improvement of practice, experimental studies and/or case studies should be conducted to identify the extent to which schools can make use of the proposed dynamic approach to school improvement.


An experimental studyexperimental study concerned with the use of the dynamic model for improvement purposes at teacher level (Antoniou, 2009) was conducted. The teaching skills, associated with the teacher factors of the model, can be grouped into five levels which are discerned in a distinctive way and move gradually from skills associated with direct teaching to skills concerned with new teaching approaches (Kyriakides, Creemers & Antoniou, 2009).Teachers situated at higher levels were found to have better student outcomes.


DEVELOPMENTAL STAGES OF TEACHING SKILLS

Levels1. Basic Elements of Direct Teaching2. Putting aspects of quality in direct teaching

and touching on active teaching3. Acquiring quality in active teaching and reaching out4. Differentiation of teaching and putting aspects of

quality in new teaching.5. Achieving quality and differentiation in teaching

using different approaches

1) Frequency management of time 2) Stage Management of time3) Frequency structuring4) Frequency Application5) Frequency Assessment 6) Frequency Questioning 7) Frequency teacher-student relation

1) Stage Structuring2) Quality Application3) Stage Questioning4) Frequency student relations5) Focus Application6) Stage Application7) Quality of questions

LEVEL 1

LEVEL 2 1) Stage student relations2) Stage teacher-student

relation3) Stage Assessment4) Frequency Teaching

Modelling5) Frequency Orientation6) Focus student relations7) Quality: feedback8) Focus Questioning9) Focus teacher-student

relation10) Quality structuring11) Quality Assessment

LEVEL 3

1) Differentiation Structuring2) Differentiation time

management3) Differentiation

Questioning4) Differentiation

Application5) Focus Assessment6) Differentiation

Assessment7) Stage teaching

modelling8) Stage orientation

LEVEL 4

1) Quality teacher-student relation

2) Quality student relations

3) Dif teacher-student relation

4) Differentiation student relations

5) Focus Orientation 6) Quality Orientation7) Differentiation

Orientation8) Quality of teaching

modelling including differentiation

9) Focus Teaching Modelling

LEVEL 5


This study attempted to find out whether providing teachers with training on their skills grouped in these five levels will help them master the skills within their level and move to the next level up (Antoniou, 2009). We randomly split into two groups the teachers who participated in this study and were found to be at a certain developmental stage. The first group was provided training by taking into account the stage at which they were situated whereas the holistic approach was used with the second group. This project shows that the use of the dynamic model to develop in-service training programmes is more effective than the holistic approach (see Antoniou, 2010)

USING AN EVIDENCE-BASED AND THEORY-USING AN EVIDENCE-BASED AND THEORY-DRIVEN APPROACH TO IMPROVE SCHOOL DRIVEN APPROACH TO IMPROVE SCHOOL EFFECTIVENESS: THE COUNTRY SPECIFIC EFFECTIVENESS: THE COUNTRY SPECIFIC

PROJECT CONDUCTED IN SLOVENIAPROJECT CONDUCTED IN SLOVENIA

To identify the extent to which the establishment of self-evaluation mechanisms cannot in itself help schools improve their effectiveness, but either the schools should make use of the knowledge-base of EER provided by the dynamic model or should first improve their climate and then build effective improvement mechanisms.

A) Main Aim of the Study

THE COUNTRY SPECIFIC PROJECT THE COUNTRY SPECIFIC PROJECT CONDUCTED IN SLOVENIA: METHODSCONDUCTED IN SLOVENIA: METHODS

A sample of 50 primary schools has been selected and achievement in mathematics and science of grade 4 students is measured using the instruments of the collaborative project (with items based on TIMSS).

The school sample will be randomly split into three groups in order to measure the effects of three different approaches of establishing SSE mechanisms on student achievement in mathematics.

Different types of support will be provided to each group of schools to establish SSE mechanisms in order to make decisions on how to improve their policy and practice.

B) Rational: Designing a multi-treatment experimental study


First group: The decision making process in establishing criteria for SSE is left to the school stakeholders (i.e., students, parents, and teachers).

According to the participatory models of evaluation, the involvement of students, parents, and teachers in defining the criteria of SSE will eventually encourage their active participation in using SSE for improvement reasons.

The first approach is related to the assumption that the school stakeholders are able to develop SSE mechanisms and define their own improvement strategies. In this way, ownershipownership is established and it is seen as a stimulus for taking responsibility for school improvement.



The second treatment gives emphasis to the establishment of a school climateschool climate that supports change.

The stakeholders of these schools will be encouraged to express their concerns about SSE and exchange them with each other.

At the next stage, support will be provided to the schools to establish their own SSE mechanisms by generating their own criteria of SSE and collecting data in relation to these criteria.



The third group will not only be encouraged to establish SSE mechanisms but also to take decisions for their improvement strategies which are in line with the assumptions of the dynamic modeldynamic model.

Beyond presenting the dynamic model and its assumptions to the school stakeholders, the instruments used to test the validity of the model at the school level will be administered.

The results of this evaluation will be presented to the school stakeholders and they will be encouraged to design their own school improvement initiatives in such a way that one of the first three priorities of their schools could be addressed.


USING THE DYNAMIC MODEL FOR IMPROVEMENT USING THE DYNAMIC MODEL FOR IMPROVEMENT PURPOSES STUDIES CURRENTLY BEING PURPOSES STUDIES CURRENTLY BEING

UNDERTAKENUNDERTAKEN

1. A study investigating the extent to which schools can use an evidence-based and theory-driven approach to face bullying is currently being undertaken in five European countries.

Main aim of this study:

To find out whether a framework based on research on bullying and on the dynamic model can help schools to establish strategies and actions to define and achieve goals associated with bullying and ultimately reduce bullying.

STUDIES CURRENTLY BEING UNDERTAKENSTUDIES CURRENTLY BEING UNDERTAKEN

2. A study investigating the impact of the dynamic approach to teacher professional development

This project investigates if the use of the dynamic model to develop in-service training programmes is more effective than the use of the holistic approach which is also advocated widely for teacher professional development. The extent to which the impact of these two approaches depends on whether they are offered externally or on a school-basis is examined. This is due to the fact that the dynamic model emphasises the relation among school level factors (i.e., policy on teaching and policy on the school learning environment) and teacher professional development.

FINAL CONCLUSIONS FINAL CONCLUSIONS

Projects investigating the impact of the dynamic approach to school improvement should be conducted.

These studies will eventually reveal the strengths and weaknesses of this approach to school improvement and may help us identify under which conditions teachers and schools can make use of this approach to improve the quality of education.

These studies may contribute in the establishment of a knowledge base of effective school improvement efforts and encourage schools to use an evidence-based and theory-driven approach to school improvement.

Thank you for Thank you for being attentive being attentive

……

Documents

USING EDUCATIONAL EFFECTIVENESS RESEARCH TO IMPROVE THE QUALITY OF EDUCATION: THE CONTRIBUTION OF THE DYNAMIC MODEL Leonidas Kyriakides Leonidas Kyriakides