Planning (and working on) school improvement

Preliminary evidence from the Quality and Merit Project in Italy

Caputo A., Mori S., Rastelli V.

Improving Education through Accountability and Evaluation Lessons from Around the World

October 3-5, 2012 - Rome

Summary

PQM Project: Training on diagnostic tools

Training on improvement interventions

Preliminary evidence on PQM effectiveness on student achievement

Aim of the studyAnalysis of School Improvement Plans characteristics/contents and their association with school variables

Results and Conclusions

The context

Lower secondary school in the Southern Italian regions having access to the European Union funds for low income EU regions (Campania, Sicily, Calabria and Apulia)

In-service teacher training

The teachers participating in the project are part of a network of schools coordinated by a tutor, who gives them both formal and online polyvalent training, all along the school year

The goals•Help teachers to set up a school Improvement Plan •Provide didactic material and alternative teaching solutions

PQM Project

The process

1. Student achievement assessment at the beginning of the school year (INVALSI pre-test)

2. Analysis and diagnosis (expert school teachers)

3. Improvement interventions (expert school teachers)

4. Student achievement assessment at the end of the school year (INVALSI post-test)

PQM ProjectTEACHER TRAINING

Training on diagnostic tools

Drafting a School Improvement Plan

Training on improvement interventions

Planning didactic activities

- remedial and extra education for students

- producing new didactic materials

- teacher peer-to-peer laboratory sessions

The core of the project

Pre-post analysis on math achievement was carried out on over 13.000 students (coming from 248 schools) participating in PQM activities in both the 2009/2010 (6th grade) and 2010/2011 (7th grade) school years.

Regardless some limitations (need of anchoring of the tests, need of repeating the analyses on control group, unavailability of more waves of data), some evidence exists on the fact that PQM students are improving.

This difference remains significant also controlling for regions, socio-economic status (SES), participation and treatment intensity (number of didactic units, school and class size, percentage of PQM students and classes out the total of the school)

Preliminary evidence

.

Pre-post analysis reveals an increase in PQM student math scores (p. < 0.01). On average students get 4 percentage points in correct answers from 2009/2010 to 2010/2011 school year.

Aim of the study

Explore characteristics/contents of School Improvement Plans

Assess the association between school planning strategies and some variable at

school level, i.e. geographical area, student socio-economic status and student math

achievement improvement

School Improvement Plan

1) analysis of the educational context. It should be addressed to both school and class levels

2) diagnosis of student needs, based on the INVALSI assessment of student achievement deficits

3) detection of improvement goals, i.e. remedy/empowerment of student education, teacher professional development and parental involving in school activities

Method

Computer-aided Text Analysis on 248 School Improvement Plans by each section:

• analysis of the educational context • diagnosis of student needs• detection of improvement goals

Lexicometric analysis

Formal characteristics of the texts in terms of lexical richness, variety, originality, level of detail

Thematic Analysis

Thematic domains and latent dimensions

ResultsTable 2 - Lexicometric Indexes of School Improvement Plans Sections

Calabria is the Region whose School Improvement Plans are generally characterized by greater richness and detail, whilst in Campania they tend to be slightly more stereotyped.

SES is not significantly associated to any measure.

The Improvement Plans of schools with very high student achievement improvement provide a more accurate analysis of the context and a greater originality and specificity of improvement goals

The Improvement Plans of schools with very low student achievement improvement show a more precise and articulated diagnosis of student needs.

Figure 1 - Analysis Of The Context. Factorial Space

F1 - School responsibility for student education Low achievement improvement External responsibility

F2 - School self-efficacy High SES High self-efficacy

Figure 2 – Diagnosis Of Student Needs. Factorial Space

F1 - Utilization of standardized test Low/very low achievement improvement Multi-focused approach F2 - Integration between internal and external assessment Low/very low achievement improvement Low/very low SES School self-assessment

Figure 3 - Detection Of Improvement Goals. Factorial Space

F1 – Specificity of planned goals Very high achievement improvement Specific goals

F2 - Originality in School Improvement Plan elaboration Very low achievement improvement Stereotypical elaboration

Conclusions

The key-factors which seem to promote better student achievements concern the school capabilities of:

Carrying out a careful analysis of the educational context

Prioritizing the various elements already in the diagnostic part of the process

Detecting specific and detailed improvement goals

ConclusionsThe main obstacles to school improvement:

Tendency to attribute responsibility for student education to external socio-educational agencies

Exclusive focus on school self-assessment to detect student needs Diagnosis of student learning deficits not based on cognitive processes or specific subject areas to enhance Poor autonomy and originality in setting up improvement activities

References

City, Elizabeth, et al. Instructional Rounds in Education: A Network Approach to ImprovingTeaching and Learning. Cambridge, Massachusetts: Harvard Education Press, 2009.

Creemers, Bert, and Leonidas Kyriakides. The dynamics of educational effectiveness: A contribution to policy, practice and theory in contemporary schools. London: Routledge, 2008.

Creemers, Bert, and Leonidas Kyriakides. “Explaining stability and changes in school effectiveness by looking at changes in the functioning of school factors”, School Effectiveness and School Improvement: An International Journal of Research, Policy and Practice 21 no.4 (2010): 409-427.

Creemers, Bert, and Leonidas Kyriakides. Improving Quality in Education: Dynamic Approaches to School Improvement. London: Routledge, 2012. Fullan, Michael. Leadership and sustainability: System thinkers in action. Thousand Oaks, CA: Corwin, Toronto, Canada, Ontario Principals Council, 2005.

INVALSI Istituto nazionale per la valutazione del sistema educativo di istruzione e di formazione, Rilevazioni degli apprendimenti. Scuola primaria. Prime analisi. INVALSI, Rome, 2009.

INVALSI Istituto nazionale per la valutazione del sistema educativo di istruzione e di formazione, Le prove del Servizio Nazionale di Valutazione 2008-2009 - Analisi Tecnica. INVALSI, Rome, 2010.

OECD Organization for Economic Co-operation and Development, Education at a Glance 2010 Revised edn. OECD, 2010.

THANKS FOR YOUR ATTENTION