30 Nymphe Street - The Evaluation Research Agency 2011_PMRP … · Web viewThe word sums in the instrument are in very simple English (e.g. Sipho has 5 books and Themba has 4 books

Eric Schollar and Associates c.c. reg. no. CK 94/36640/23 VAT reg. no. 414 014 7069

THE ZENEX FOUNDATION

BASELINE STUDY OF THE EVALUATION OF THE PRIMARY MATHEMATICS RESEARCH PROJECT UNDER THE MANAGEMENT OF THE LIMPOPO

DEPARTMENT OF EDUCATION IN VHEMBE DISTRICT

Eric Schollar6 April 2011

30 Nymphe StreetKensingtonJohannesburg 2094South Africa

Phone +27 (0) 11 622 7075Fax +27 (0) 11 622 7589Cell +27 (0) 83 255 0109Email [email protected]

CONTENTS

History and objectives of the PMRP in Limpopo ………………………………………………. 1Transfer of the programme to the Limpopo Department of Education ………………………… 1Evaluation of the programme under departmental management ……………………………….. 3Findings: Action-research: Development and outcomes of the transfer process ………………. 4Findings: Status of the programme in schools at June 2010 …………………………………… 9Impact evaluation study design ………………………………………………………………… 11Longitudinal tracking of the field trial cohort: 2007 to 2010 …………………………………... 14Baseline data: 2010 ……………………………………………………………………………... 16Years two and three of the evaluation …………………………………………………………. 19Appendix One: Establishing the formal status of the PMRP in Limpopo ……………………… 20Appendix Two: Steering Committee: Agenda for handover meeting and participants ………… 21Appendix Three: Programme notes for management and quality support group ………………. 23Appendix Four: Monitoring information by circuit and school ………………………………... 35Appendix Five: Numbers of schools and learners tested by circuit ……………………………. 36Appendix Six: Item analysis of instruments ……………………………………………………. 39Appendix Seven: Raw scores and percentages by circuit ……………………………………… 41

ESA

Baseline Study of the Evaluation of the PMRP under Departmental Management

1. History and Objectives of the PMRP in Limpopo

The Primary Mathematics Research Project was initiated in 2004, with the endorsement of the National Department of Education. Its brief was (1) to research the causes of the persistence of very poor outcomes in mathematics in South African primary schools - despite the introduction of the new curriculum in 1998 and the ever increasing allocation of state revenue to education, and (b) to develop a developmental programme in response to these findings. The Limpopo Department of Education provided its support in 2007 to the field testing of the programme developed by the PMRP, locating it as a research project of the department within the provincial Learner Attainment Strategy under the direction of the Senior General Manager: Curriculum.

Subsequent to the significant improvements in learner performance measured during the field trial, ESA was invited in November 2008 to present a proposal to the Executive Management Meeting of the LDoE for the piloting and researching of the most effective available methods of transferring the operational management of the PMRP to departmental officials. The underlying intention was to examine the feasibility of potential large-scale replication of the PMRP programme in the province and, as such, the PMRP remained a research project of the department within the Learner Attainment Strategy under the direction of the SGM.

The two essential objectives of the programme that resulted from this proposal are to: maintain the same, or similar, levels of impact on learner performance as those measured in the

20 schools in the project group during the field test on a much larger scale – i.e. in 125 schools. transfer the routine operation and management of the programme to the Limpopo Department of

Education over the first one to two cycles of programme implementation, training and mentoring without significant loss of programme quality and, hence, lower levels of impact on learner performance.

The proposal also provided for a partnership between the LDoE, ESA and private sector agencies in researching and piloting a workable transfer strategy: The first cycle of delivery of the PMRP would be implemented and managed by ESA through

funding from the external partners – in this case, the Zenex Foundation, the Anglo American Chairman’s Fund and XSTRATA South Africa. The external funding would provide for training of departmental officials, SMT members and teachers, as well as a complete supply of Teachers Manuals, Learner Workbooks and Diagnostic Tests for each participating school; Grades 4 to 6.

At the end of each first cycle of delivery, operational management of the programme would be handed over to the Limpopo Department of Education – and the department would supply the Learner Workbooks for incoming Grade 4 classes required to sustain the programme during delivery Cycles Two and Three.

2. Transfer of the Programme to the Limpopo Department of Education

Transfer of the operational management of the PMRP programme was completed by December 2011 and the PMRP is currently operating in the whole of the Malamulele Cluster in the Vhembe District of Limpopo – in Malamulele North East, East, West, Central and Vhumbedzi Circuits. Delivery in 2010 commenced on 14 February with the delivery and distribution of Learner Workbooks and Teacher Manuals, training commenced on 18 February, and school support/mentoring visits on 7 March. Participating in the programme there are: 125 schools 370 teachers Over 20 000 learners 5 Circuit Managers

1

ESA


125 Principals/SMT members 3 Vhembe District Mathematics Advisors 5 Circuit PMRP Teacher Committees 1 Area PMRP Teacher/SMT Committee

The original transfer strategy centred on the department’s district advisory service as the logical institutional locus-of-control but this proved unworkable; there were only three GET mathematics coordinators for the 27 circuits and 692 schools in Vhembe! Nonetheless, all three District Curriculum Advisors - one of whom is the Vhembe GET Mathematics Coordinator and the DoE PMRP Project Manager, as well as the District FFLC Coordinator – were trained and carried out school support and monitoring visits during the first and second terms of 2011. One has since retired and has not yet been replaced.

The adapted transfer strategy was shifted to focus on local-level structures and individuals to provide operational management after the withdrawal of ESA. These structures and individuals were identified and developed by the departmental Project Manager and by ESA 2010 and 2011; ‘formal’ handovers to these structures were made in October 2010. The programme is currently managed by a three-tier structure: The Vhembe Steering Committee comprised of the District Curriculum Coordinators, Circuit

Managers and a Principal/AMESA representative. See Appendix Two for the Agenda of the institutional transfer meeting on 5 October 2010, along with a list of the members. This is primary an ‘enabling’ structure.

The PMRP Area Committee comprised of two Principal/SMT members and two teacher members from each of the five circuits, together with the circuit managers and the departmental Project Manager of the PMRP. This is primarily a representative forum and communication structure.

A Management and Quality Support Sub-committee of the Area Committee comprised of two leader-Principal/SMT members and two Key-Teacher members from each of the five circuits. This is primarily a programme management and support structure. See Appendix Three for the training notes provided to this sub-committee at the operational transfer meeting on 27 October.

2.1. Relationship of the PMRP to Other Departmental Programmes

The PMRP programme is conceptually and operationally integrated with national and provincial programmes.

It is effectively a successor programme to Khanyisa in Vhembe, especially in terms of the Limpopo Numeracy Strategy, and much of its design was influenced by ESA’s participation in and review of Khanyisa.

With regard to the Foundations for Learning Campaign, the PMRP programme supports the fundamental objectives of the FFLC by providing a solution to the universal problem of ‘multi-grade’ classes in the great majority of our schools in which the actual ability level of many learners can be two, three or even four grade levels below the minimum expected. It provides a ‘catch-up’ programme starting with diagnostic testing to support the acquisition of competencies learners have failed to grasp in earlier grades. Only once learners have mastered missed content can they be expected to successfully deal with the correct Assessment Standards for their grade level – the ‘core’ objective of the FFLC.

2

ESA


Dr. Sambo of the LDoE has suggested the introduction of the programme to Dinaletsana schools. Preliminary discussions have started and ESA has offered to provide support to the implementation of the programme in these schools/circuits using the same transfer model currently applied.

3. Evaluation of the PMRP under Departmental Management

The Zenex Foundation approved a proposal for the evaluation of the PMRP and its associated transfer strategy commencing in Term Three of 2010. The summative quantitative impact component of the study is based on longitudinal tracking of the 2010 Grade 4 classes in all 125 schools from entry into the programme to exit at Grade 6 – the data will provide very reliable information about nature and degree of impact on learner performance during the period that the programme is operated by the department itself. This is a potentially ground-breaking study in South African mathematics education and institutional development.

The degree of impact on learner performance will be dependent on the coherence, maintenance and effectiveness of programme delivery in classrooms by the current institutional (departmental) arrangements.

Figure 1: Relationship between strategic approach and delivery system in achieving impactsStrategy on which intervention is basedEffective Ineffective

Delivery of intervention Effective Predicted impact likely Predicted impact unlikelyIneffective Predicted impact limited Predicted impact unlikely

The overall process or operational objective of any intervention is to ensure that BOTH the strategic/theoretical underpinning AND the delivery of the intervention are effective. Many of the larger scale interventions in recent years have shared the idea of a systemic intervention design operating, to one degree or the other, across all levels of the school system; province, district, circuit, school, classroom and community. IMBEWU in the Eastern Cape was one of the earliest and KHANYISA in Limpopo one of the most recent – EQUIP of the NBI, is another example, as are the DDSP and IEP of USAID.

In all of these interventions, the provincial departments of education lacked the capacity to manage and deliver the intervention design without external assistance and extensive use was made of external agencies, especially NGOs, to provide field staff to the programme. Attempts to build capacity in education departments during the period of intervention delivery were almost always based on the training and mentoring of officials by the external ‘co-facilitators’ – as they were termed in Khanyisa. It was assumed that these departmental officials would participate consistently, learn ‘on-the-job’ and be able to assume operational management of the programme once the intervention itself came to an end and the external facilitators withdrew. They would also learn how to maintain and improve the quality of the delivery of the programme at classroom level. In the event, this proved much more difficult to achieve than expected. Instead, officials and teachers alike tended to regard the interventions as limited-term NGO ‘projects’ that had a discrete life. External NGO facilitators were generally left, in practice, to deliver the programme on their own but typically lacked the authority required to really establish coherent and organized programmes of instruction in schools. The consequence is that most of these interventions failed to achieve systemic change in ‘lodging’ the programme within the routine operations of the departments, schools or teachers concerned. This is not to say that none of these interventions achieved change, or that no lasting effects can be discerned, but to recognize that enduring systemic and organizational change was not achieved as intended by the programme design.

3

ESA


This has been the central operational challenge for the PMRP since its inception in Limpopo. Consequently, and at the express instruction of the SGM: Curriculum, the programme has always been accompanied by a strong emphasis on action-research; to explore, identify and develop the capacity of an education department to establish as a routine element of its regular functioning, an intervention programme without the need for long-term external support. ESA field researchers have visited all of the schools/teachers three times in the original pilot districts (Malamulele North East, East and Central) and twice in the 2010 circuits (Malamulele West and Vhumbedzi); 323 school visits over 2009 and 2010. At each visit, the SMT was interviewed as well as all of the Grade 4 to 6 teachers; lessons were observed on each visit. Regular contact with Circuit Managers (some are more ‘active’ than others) has been maintained by ESA and the Project Manager, along with key SMT members and teachers as they emerged. The early meetings of the Area Committee focussed on participants themselves defining the processes they need to develop; the emergence of the Management and Quality Support sub-committee is the best example of the result.

Also significant is that the Area Committee established an annual Mental Maths Competition in support of Foundations for Learning for all PMRP schools in 2010. The members of the Quality Support sub-committee set the test instruments, organized the event, helped coordinate transport and so on.

The culmination of the action-research process was the development of a set of constraints to transfer, together with a proposed protocol for dealing with them, that was submitted to the SGM: Curriculum, as well as the Steering and Area Committees. Also developed was a monitoring system aimed at providing basic programme information to management; its initial focus has been to place each of the schools in one of three categories. Both of these developments are reported in the following section.

This section makes extensive use of the reports that ESA submitted to the department, and to the ZF, at the end of Phase IV (the transfer phase) in June 2010. To illustrate the ‘sequential’ and developmental nature of the operational process with which they were dealing, tenses are left as they were in the originals. Combined with the monitoring information, these reports constitute the formative, contextual, managerial, programmatic and qualitative components of the evaluation.

3.1. Findings: Action-research: Development and Outcomes of the Transfer Process

3.1.1. Constraints to Transfer in June 2010

There have been five major constraints to the overall objective of the transfer of the operational management of the PMRP to the Limpopo Department of Education.

Constraint One: Supply of Learner Workbooks for 2010 Grade 4 LearnersThis was always seen a key component of the transfer strategy. The SGM duly provided formal authorization in time for the materials to be delivered at the start of Term Two. However, the order has still not been processed (i.e. by June 2010). In the meantime, some schools have been using old workbooks and others photocopied material, while others have simply stopped implementing the programme at Grade 4 level.

Constraint Two: Pending Retirement of Curriculum AdvisorsThere are only three Curriculum Advisors for the 27 circuits in Vhembe. Nonetheless, all of the advisors were trained and participated in school support and monitoring visits during Term One.

4

ESA


Again, this was always seen a key component of the transfer strategy - however, two of these advisors are retiring at the end of the year and are now confined to the Thohoyandou offices and are no longer available for school support and monitoring. We are not aware when new advisors will be appointed.

Constraint Three: Failure to Continue the Use of the Programme by Some TeachersSome of the 2009 PMRP-trained teachers, and some of the 2010-trained teachers, have stopped implementing the programme as the transfer accelerates, effectively assuming that the PMRP is a NGO ‘project’ which is ‘voluntary’. ESA field researchers do not have the authority to insist on compliance and SMT members as well as Circuit Managers are disempowered by the lack of a formal MOU between PMRP/ESA and the LDoE.

Constraint Four: Replacement of all Mathematics Programmes by Assessment Resource BanksMany teachers continue to believe that the Assessment Resource Banks, distributed through the Khanyisa Programme, are the ‘official department programme’ and have been using them as a sort of ‘text book’ through which they now only teach solutions to problems contained in the ARB. Consequently, organized mathematics education has almost completely collapsed in these schools while teachers drill learners through the ARB task-by-task. The misunderstanding of the ARB has been very strongly influenced by the continuing use of ARB tasks verbatim in Common Assessments. Any apparent gains in learner performance achieved by this practice will, obviously, disappear when learners are tested on instruments that are not lifted from the ARB.

Constraint Five: Understanding of Relationship between the PMRP and Foundations for Learning CampaignMany teachers regard the FFLC as the new prescriptive policy and that they are not allowed to use any other materials despite the assurance to the contrary contained in the FFLC files. Consequently, while their learners struggle with problems based on Assessment Standards well beyond their level of competence, teachers neglect to use the PMRP programme designed to deal with this very problem. This frustrates the fundamental objectives of both the FFLC and the PMRP – not to mention the Learner Attainment Strategy.

3.1.2. Proposed Protocol for the Transfer of the PMRP Programme to December 2010

The PMRP has reached a critical point in piloting the transfer strategy defined in 2009. Training is almost complete, all of the necessary local-level structures have been developed and roles for participants defined. The programme plan calls for the withdrawal of ESA field researchers from the Malamulele Cluster at the end of Term Two but it is clear that the constraints noted above will significantly reduce, if not entirely destroy, the continued operation of the programme by the department unless they are directly addressed.

The programme is in other words, in some danger of collapse and failure to achieve the objectives agreed with the department and with the funding partners. Our experience, and that of the DoE PMRP Project Manager, has convinced us that a number of actions are necessary to ensure that this collapse does not occur, and that the LDoE does not lose the benefits we have achieved thus far – as well as those possible in the future. If the programme collapses, the department will still be left without an effective, evidence-based and diagnostic response to the crippling problems created by multiple levels of learner performance at each grade level that make the likelihood of achieving the provincial targets set by the Foundations for Learning Campaign very low indeed.

Formalize the Status of the PMRP Programme

5

ESA


The PMRP has operated as a departmental research programme, supported by a broad partnership, within the Learner Attainment Strategy but a formal MOU was never signed, and nor were participants in the Malamulele Cluster formally notified by province and district that the PMRP was an official departmental programme. Consequently, because of the direct involvement of ESA, many local-level participants still regard the programme as a ‘voluntary’ NGO-type project that is ‘about to end’. It is now necessary, in terms of the transfer strategy, to formally indicate that the PMRP is an operational programme of the Limpopo Department of Education in the Malamulele Cluster. Only once participants are assured of this will they routinely continue to operate the programme in classrooms once ESA withdraws. Equally, and very importantly, SMT members and Circuit Managers are currently disempowered in terms of monitoring the programme and ensuring its continued implementation. Only if they are informed that the PMRP is, indeed, a departmental programme will they be able to provide the operational management required to secure effective transfer.

Formalize the Status of the DoE Project ManagerThe Project Manager – the single most critical participant in the transfer strategy - has been able to operate thus far without formal appointment by province and district. This is becoming untenable as complete transfer looms and other officials, SMT members and teachers question the source and extent of his managerial authority once ESA withdraws. Consequently, it is necessary to provide him with a formal appointment that will enable him to deal correctly with departmental protocol, planning and work scheduling – for one thing, he needs to be able to factor PMRP management into his own work plans and time allocation. It would also be necessary to provide him with a small administrative budget – ESA has provided him with a laptop computer, 3G internet access and printer/scanner without charge to the department but he needs to purchase replacement ink cartridges and paper out of his own resources.

Provide Curriculum Advisors with Transport to Enable School Support and MonitoringThe training and direct participation of Curriculum Advisors in school support has been a significant achievement of the PMRP in terms of the transfer strategy. However, pending retirement they do not have access to department transport and are currently confined to the Thohoyandou office. I understand that it is possible to allow them a subsidy to use their own vehicles to resume school support and monitoring visits. In addition, they should be tasked with providing orientation and training to the newly appointed advisors - if they are appointed before the end of 2010.

Secure and Rationalize the Supply of Learner Workbooks to Grade 4 LearnersThe non-arrival of the workbooks for the Grade 4 learners in Malamulele East, North East and Central has had a significant negative impact on the operation of the programme and on the perceptions that teachers hold of the PMRP. Assuming that the department wishes to continue with the programme, these books should be supplied as soon as possible. In addition, plans should be made before the end of 2010 to provide books to the incoming Grade 4 learners in 2011 at the start of the first term.

Address the Issue of the Assessment Resource BanksWhether the department continues to use the PMRP or not, it will still be necessary to take vigorous action to ensure that (a) officials NEVER lift items verbatim from the ARB when setting Common Assessments, and that (b) teachers immediately cease working their way through the ARB item by item. If this is not done, it is reasonable to fear that mathematics education will collapse completely in schools and that learners will fail to show any gains on objective test instruments.

Confirm the Relationship between the FFLC and the PMRP

6

ESA


District, Circuit and school-level participants should be informed of the nature of the relationship between these two programmes. In short, the PMRP is intended to make possible the achievement of the fundamental objectives of the FFLC, especially with regard to Learning Outcome One. The PMRP programme covers all of the Assessment Standards of this LO, so teachers using the materials will be meeting the requirements of both the NCS and the FFLC for this crucial LO. They must, of course, continue to deal with the other Learning Outcomes and it is self-evident that improved learner performance in LO1 will greatly support their ability to do so.

We are testing the ability of the PMRP programme to support the development of the competencies learners require in order to meet the objectives of the FFLC so it is necessary to ensure that schools apply the complete programme. If we achieve this objective, learners in PMRP schools should be significantly closer to the standards required by the FFLC, especially once they have been exposed to the programme from Grade 4 to Grade 6 – this would be a genuinely significant achievement for both the LDoE and the PMRP.

Provide Formal Notification of the Evaluation StudyThere is a need to ensure that participants understand the evaluation that will commence in Term Three of 2010, and will continue until 2012, has the endorsement of the department.

Support the Appointment of Key Teachers to Sustain Programme QualityWith the current non-activity and pending retirement of the Curriculum Advisors there is a very great need for a strategy to provide continuing support to the maintenance of programme quality in schools once ESA withdraws. The exiting PMRP Area Committee has already provided significant collegiate support to the programme in schools, including organizing an annual Mental Mathematics Competition, and has helped identify, together with ESA, potential Key Teachers to receive supplementary training from ESA.

This will provide the department with a significantly increased capacity to provide quality support to schools and teachers. It is suggested that these teachers (2/circuit – 10 in all) be provided with afternoon training workshops combined with experiential on-site mentoring by ESA. The on-site training/mentoring would necessitate only a few days release of these teachers from normal duties before they would be able to deal with situations and queries that arise during the practical application of the programme – for example, say 3 days over a two month period. After training, the Key Teachers would be available to the Project Manager, Circuit Managers and Curriculum Advisors; their activity would largely be focussed on providing after-hours cluster workshops to teachers as issues arise. They would also be a central resource in training teachers newly appointed to the schools.

3.1.3. Outcomes of Proposed Protocol

Formalize the Status of the PMRP ProgrammeFormalize the Status of the DoE Project ManagerProvide Formal Notification of the Evaluation Study

All three of these objectives were achieved through a circular sent by the SGM: Curriculum to the Vhembe District Senior Manager, the Vhembe GET Mathematics Coordinator and the five Circuit Managers. See Appendix One for a copy of this circular.

Provide Curriculum Advisors with Transport to Enable School Support and Monitoring

7

ESA


This objective was not achieved – and there are only two mathematics Curriculum Advisors in Vhembe in 2011.Secure and Rationalize the Supply of Learner Workbooks to Grade 4 Learners

This objective was achieved in 2010 when workbooks were provided through the Learner Attainment Strategy in October. They are currently with the Grade 5 learners in 2011 and, therefore, we will still be able to continue with the longitudinal evaluation study. Discussions have commenced for workbooks for Grade 4 in 2011 and we are still waiting to hear the outcome (March 2011). In the meantime, schools have been asked to use photocopies of exercises for Grade 4 learners.

Address the Issue of the Assessment Resource BanksConfirm the Relationship between the FFLC and the PMRP

Both of these objectives were achieved in the sense that department officials have conveyed the appropriate information to PMRP schools – it remains to be seen how the situation develops in 2011.

Support the Appointment of Key Teachers to Sustain Programme Quality

This objective was achieved through the agreement of the Circuit Managers. The Management and Quality Support sub-committee is made up of Key Teachers and Key Principals/SMT members.

3.1.4. Conclusion

In support of the original objectives of the Zenex Foundation and of department for Phase IV of the PMRP in the Malamulele Cluster, ESA continued to provide field researchers to the department during the third and fourth terms - despite the fact that we had no budget for doing so. The presentation of the supplementary training workshops to Key Teachers, and the Area Committee in general, combined with ‘on-the-job’ mentoring, were seen as critical in making programme handover sustainable. I believe that Appendix Two reflects the progress we have made in this regard. In addition, Ms Gugu Zulu of the Foundation has attended meetings of both the Steering and Area Committees and has seen, at first hand, the commitment and enthusiasm of many of the members.

While the outcome of the pilot of the transfer strategy of the PMRP is not yet certain, one way or the other, we have already learned a great deal about the practicalities of institutional development, and mathematics education, that will be available in future to the LDoE and to funding partners in future. In this sense, the research programme has already been a success - and the evaluation of the operational management of the PMRP by the department promises to extend the findings of this research, one way or the other.

In short, I think that a great deal has been achieved to date and that it is still too early to conclude that the current transfer strategy will prove unworkable; it has continued to evolve in response to realities and has become ever more comprehensive, operationalized and formalized.

3.1.5 Current Situation: March 2011

The Departmental Project Manager has reported that the Management and Quality Support sub-committee organized the scheduled school meetings in December 2010. In 2011, he has received reports from sub-committee members that the programme is being implemented in most, if not all,

8

ESA


of the schools. The Project Manager has also commenced school monitoring visits and is able to verify this situation in the schools he has visited – his report has already been submitted to the ZF. In my view, the most important points he makes are:

The majority, at least, of schools continue to implement the programme in all five circuits Secondary school principals and teachers confirm the improved quality of learner

performance of learners coming from primary schools – especially in terms of improved accuracy and fluency of calculation, written and mental, at the expected Assessment Standards. (This comment suggests further research that could provide some interesting information.)

The level of motivation on the part of both teachers and learners towards providing effective teaching and learning of mathematics is improving in PMRP schools

The PMRP programme has proved to be an effective means of supporting the implementation and impact on learner performance of the Foundations for Learning Campaign.

ESA researchers will visit the area to verify the extent of programme delivery, and provide training/support to the Management and Quality sub-committee as required, between end-April and mid-May. This is not a budget cost for the Foundation and nor does it form part of the Phase IV or evaluation project proposals.

3.2. Findings: Status of the Programme in Schools at June 2010: Monitoring Information

This information is derived from the data base developed by ESA over the first half of 2010. The intention was to pilot a method of collecting and providing useful information to guide project management by the department through the Project Manager and the Quality and Management sub-committee. ESA field researchers collected the data presented below and have assisted in refining the monitoring instrument – the figures provided below are, therefore, from the pilot version but demonstrates the kind of information that is/will be available to project management.

Schools are placed into one of three categories - the aim of management is to get all schools into Category One:

1: Implementing the programme at a sufficient pacing and level of quality2: Implementing the programme but need support in increasing pacing or improving quality3: Either not implementing the programme, or are doing so with a negative attitude and low level of quality

It immediately became evident that teachers in one school can be very different, especially in terms of attitudinal and motivational levels. We assumed that the standardization of the programme and its management would operate across schools irrespective, to a limit, of the inherent quality of the teachers at each school and this improved to be generally true but we underestimated the problems of generally negative attitudes and low levels of motivation in some teachers.

Consequently, in one school a teacher can be enthusiastically running the programme very well while another pays little more than lip-service to doing so. The final version of the instrument will be based on individual teachers rather than schools per se.

9

ESA


Table 1: Percentages of schools in each status category by circuit

Circuits One Two ThreeCentral 20 40 40East 33 42 25North East 19 41 40West 26 57 17Vhumbedzi 0 54 46

Vhumbedzi is a remote circuit in the mountains through which the Luvuvhu River flows in Venda; it has received very little, if any support over recent years and many of the teachers are struggling with the meaning of the curriculum and its associated methodologies. The other circuits also include many schools in remote areas but they are generally closer to larger villages, or on transport routes, while a few are in Malamulele itself or in relatively ‘urbanized’ Saselamani.

There is a generally higher level of awareness and capacity in dealing with the curriculum in these circuits, though only around 20% to 30% can be said to be truly proficient – and they are generally also the teachers in Category One who are having little difficulty with the PMRP programme in either motivational or proficiency terms. It is from this group that the Key Teachers have been selected/nominated and some of them provide impressively coherent instruction to their learners under the most difficult of circumstances - like summer temperatures that frequently exceed 40 degrees for long periods. These teachers also tend to be overburdened with other managerial and community responsibilities, as is the way of the world, but they are capable of using the programme – given a few tweaks here and there – on a self-sustaining basis.

The largest group of teachers falls into Category Two; they generally have sincere and positive motivations, and are willing to work consistently with the programme, but lack confidence or expertise and require more support in understanding and applying the different components of the programme. There is little doubt that the vast majority of these teachers are capable of achieving Category One status if they receive sufficient planning and programme direction and support; the intended role of the Quality Support Group.

Around a third of the teachers, however, fall into Category Three; they appear to generally have a negative attitude to their profession and, often, their employer – a situation worsened by the teacher strike which affected the whole of Term Three. Some of these teachers pay a sort of ‘lip service’ to the programme (and, we are told by members of the Quality Support Group, to the rest of their teaching responsibilities). The difficulty of managerial supervision in these circuits, given their size and remoteness, in the absence of sufficient numbers of district and circuit officials provides these teachers with considerable latitude – a latitude extended by the belief that teacher unions will defend any behaviour (a view not necessarily shared by the local-level leadership of any of the teacher formations!) Whatever the cause, it is evident that some teachers require the managerial authority of the department to demand sufficient compliance with their job responsibilities to the point that any form of presenting a coherent and organized course of instruction to learners in possible. This is not a situation unique to Vhembe.

10

ESA


Progressing well, 20%

Needs support to quality, 47%

Needs management intervention

33%

Figure 2: Status of programme in all circuits: % of schools

Monitoring information by circuit and individual schools can be found in Appendix Four.

4. Impact Evaluation Study Design

The fundamental objective of the study is to measure the level of impact of the PMRP programme on learner performance now that ESA has withdrawn from the field and the management of the programme handed over to the Limpopo Department of Education.

Unlike the 2007 field trial in which use was made of a randomized experimental design, the current impact study of the programme under departmental management cannot make use of a control group; because all of the schools in the five circuits are participating in the PMRP.

The literature recommends the use of very large samples and multiple measurement points in situations where a sensible control group cannot be selected. The main sources of variation in learner performance, other than intervention effect, are the varying contexts and qualitative variables existing in each school (e.g. level of staff alignment, level of school functionality, commitment of teachers, and so on). Therefore, the sample of schools needs to be high enough (i.e. have a low sampling error) to allow the ascription of change to intervention effect rather than to idiosyncratic differences in schools in the sample. The literature recommends the use of 114 units in a sample when the population is 124 for 2% precision at 95% certainty; we instead used the whole population of schools as this level to eliminate sampling error altogether. It should be noted that one of the schools is a special school that has been included in the programme but was excluded from the testing.

There are just over 20 000 learners in the 124 schools. At the level of the individual learners there are, of course, also possible sources of variation in performance other than intervention effect that must be taken into account. However, since the population of learners is very much higher than that of schools, the literature recommends a sample of just over 33% for a population over 20 000 to provide data with precision of 1% at 95% confidence – around 6 600 learners was the ideal target. To reach this sample size, we opted to select 55 learners at each school – a working target of 6 820 learners; we chose the higher figure to allow for (a) annual attrition - probably around 10% - through absence or transfer of the longitudinal cohort starting at Grade 4 and (b) the fact that some of the schools are junior or senior primaries that do not have classes after, or before, Grade 3. In the event, 6 080 learners were tested; providing a precision of just over 1% at 95% confidence.

11

ESA


Three measurement points; pre, mid and post were to be applied to three components of the sample:

40 randomly selected learners from each school from Grade 4 were tested to provide the baseline for the longitudinal cohort that will be tracked to Grade 6 – this component of the study can continue into 2011 and 2012 as the learners have workbooks.

15 randomly selected learners from each school that was not in the project or control group of the 2007 field test from Grade 6 were tested to provide:

o The baseline for the different groups of learners who will enter this grade each yearo The baseline for a proxy control for the cohort that will reach Grade 6 by post-testing

This will yield two impact measures. The first tracks the performance of the same cohort of learners from Grade 4 to Grade 6. The second measures the effect on ‘standards’ at Grade 6 level of the programme – the post measure in 2012 will be taken from a different group of learners who were in Grade 4 in 2010 (i.e. for the complete programme).

In addition, the same instruments were used in both the 2007 field trial and the current evaluation. Consequently, we were provided with an ideal opportunity to re-test the same learners who were in the Grade 4 project and control schools during the 2007 Field Trial. This cohort of learners had provided strongly significant evidence of improved learner performance in the post-tests at the end of the (14 week) field trial in 2007 after which they received no further support from ESA. To my knowledge, only one other intervention programme, NGO or academic-based, has been able to apply a second post-test - the RTI retested one cohort one year after the end of the IEP programme. The PMRP data, three years after programme-end is, therefore, unique in South African educational research.

The consequence is that, in the 38 cohort schools from 2007, learners in the Cohort (i.e. were in Grade 4 in 2007 and could be expected to be in Grade 7 in 2010) were tested first. Only if there was enough space in the test room were Grade 6 learners at these schools also tested but, in some cases, there were too many learners from the Cohort to allow testing of all, or even any, of the Grade 6 learners.

4.1. Baseline Testing

The test schedule was completed in full by the first week of Term 4 in 2010 in 124 schools in the five circuits to a total of 6 080 learners; as noted above, some of the schools are Junior Primaries and do not have Grade 6 classes, others are Senior Primaries that do not have Grade 4 classes. Small schools did not have as many as 40 Grade 4 learners, while others had more; when the number exceeded 40 at any school, 40 were randomly selected for testing. The detailed numbers of learners tested by Grade or Cohort, circuit and school are reported in Appendix Five.

Table 2: Actual numbers of schools and learners testedSchools Learners

Grade 4Malamulele Central 27 758Malamulele East 21 726Malamulele North East 24 859Malamulele West 17 653Vhumbedzi 22 732Whole sample 111 3 728

12

ESA


Grade 6 Schools LearnersMalamulele Central 9 136Malamulele East 10 150Malamulele North East 15 227Malamulele West 17 264Vhumbedzi 22 329Whole sample 73 1 106

Project Control TotalSchools Learners Schools Learners Schools Learners

2007 Cohort (Grade 7)Malamulele Central 10 385 8 259 18 644Malamulele East 5 171 7 210 12 381Malamulele North East 5 116 3 105 8 221Whole sample 20 672 18 574 38 1 246

4.2. Instruments

The test instruments, standardized for Grades 4 and 6 levels, were constructed/adapted from three sources:

Items dealing with LO1 from the National Systemic Evaluation instrument were used with the permission of the National DoE (Part One of both instruments)

Items used in the 2007 study, but standardized for the respective Grade levels, were used in Parts Two (arithmetic operations) and Four (Word Sums), as well as and most of Part Three (other non-arithmetic operation items for LO1). A couple of new items were also added to Part Three for the Grade 6 instrument.

New items were used for Part Five (arithmetic operations standardized for Grades 4 and 5) to provide an indicator of impact at these levels for Grade 6 learners

Table 3: Item distribution in test instrumentsn of items

Part Description Grade 4 Grade 6One LO 1 items from the National Systemic Evaluation instrument 12 16

Two Items dealing with the four operations from the instrument used in Phase I - but standardized for these grades 20 20

ThreeOther LO 1 items (counting, shapes, fractions, decimals conversions, relationships, sequences, factors) from the Phase I instrument - but standardized for these grades

9 20

Four Word sums (matched with operations from Part Two) 8 8Five Operations based on Assessment Standards for Grades 4 and 5 n/a 20Total 49 84

4.2.1. Item Analysis

Item analysis is usually used to help select items for test instruments in order to achieve a specific objective; in the current study it was carried out only after the baseline data had been obtained.

The instruments were specifically standardized against the expected/predicted learner performance levels stated in the National Curriculum Statement for Grades 4 and 6. However, we already knew that over 80% of learners are below their expected levels of proficiency for the grade at which they are enrolled (NSE, SACMEQ, TIMSS); the figure for mathematics in Limpopo is just over 90% -

13

ESA


and over 50% of Grade 6 learners are at a Grade level proficiency of Grade 3 or below! Therefore, using item analysis to produce an instrument easy enough to conform to ‘normal’ difficulty levels for a Grade 6 instrument would necessarily involve using many items based on Grade 3 Assessment Standards, and progressively less items from Grades 4, 5 and 6.

Since the objective of the PMRP is to ultimately improve learner performance to its expected standards, this would not have been appropriate for instrument design – resulting in an instrument on which it would have been relatively much easier to measure significant impacts on learners. Secondly, the existing instruments on which the final versions were based had already collected valuable data and the comparative value of these data also argued for retaining the items they contained, especially the NSE LO1 items where ‘simplifying’ would have been misleading. The only compromise has been to include 20 items in the Grade 6 instrument that are based on the Assessment Standards for Grades 4 and 5 – 10 items each.

The table below summarizes the outcome of the analysis for the two instruments; more detail is provided in Appendix Six while item by item information can be obtained from ESA.

Table 4: Item analysis of test instruments

Actual % of items

Difficulty index Suggested % of items Grade 4 Grade 6

> 75 25 55 4825 - 75 50 33 49

< 25 25 12 4

100 100 100%

The disjunction between expected and actual performance levels of learners on standardized instruments is immediately apparent – whereas the recommended proportion of items at a difficulty level of over 75 (on a scale of 100) is 25% both Grade scores are close to double that; the other end of the scale is equally unbalanced – there are much fewer ‘easy’ items with a difficulty level of below 25 than suggested/expected. Only in the mid-range, with a difficulty level of between 25 and 75, is the actual proportion of items almost exact for Grade 6, and somewhat closer to the expected for Grade 4.

It is not, therefore, surprising that the mean scores obtained by learners in both grades are so low – 33% for Grade 4 and 29.5% for Grade 6 – one could reasonably expect a mean on a standardized instrument of around 55% to 60%, certainly over 50%! Nonetheless, the item analysis indicates that the instruments are capable of providing reliable longitudinal data about learner performance; there are many items available to measure improvements in performance. Of particular interest in this regard are the 20 items in Part Five of the Grade 6 instrument that allow measurement of improvement of learners towards the expected standards, even if they do not achieve Grade 6 standard.

5. Tracking of the 2007 Grade Four Cohort

The intention was to measure the degree to which improved performance measured during the field trial would be maintained in the project group three years after the withdrawal of ESA and, in so doing, examine the question of the causation of the impact figures obtained in 2007. Was the impact due to improved school and teacher functionality caused by being part of the project group and

14

ESA


subject to monitoring visits by ESA field researchers? Or was it due to the learning theory, and approach to Curriculum Management, embedded in the programme?

Since the 14 weeks of the Field Trial in 2007, schools and teachers in the project group have received no further exposure to the programme through ESA and so we can conclude that the only difference between learners in the project and control groups is that the former completed at least 11 weeks (80%) of the programme in 2007. Any residual difference between the two groups is, therefore, caused by exposure to the learning theory, and approach to Curriculum Management, embedded in the programme - and has since persisted in learners.

Figure 3: Comparison of scores of project and control groups between 2007 and 2010

pre post 1 post 20

5

10

15

20

25

30

35

40

45

project control

14 weeks 3 years

Three years (2007 to 2010) after the end of the intervention, the control group has still not reached the same level of learner performance as the project group after 14 weeks of exposure to the intervention programme in 2007.

On the one hand, the gradual convergence trend in the data may simply indicate that eventually the difference will be eroded unless learners – now exposed only to ‘routine’ schooling - continue their involvement with the approach and methodology of the PMRP. On the other hand, it may be that the residual significant difference still persisting between the groups indicates the degree of impact due to the content of the intervention – the PMRP programme – and that this difference will persist; only further tracking of these learners into secondary schools could answer this question. One way or the other, however, it is clear that the impact measured during the 2007 study was not due exclusively to increased school/teacher functionality as a sort of Hawthorn Effect of the intervention – critically important as it is to achieve such increased functionality during intervention programmes.

Looking at the figures in a crude sense, it might be speculated that about two-thirds of the 2007 impact could be ascribed to these functionality effects and around one third to exposure to the learning theory, and approach to Curriculum Management, of the PMRP programme. Only further research could clarify this speculation.

15

ESA


The raw figures, rounded to one decimal, are provided in the table below – impact is 5.96% without rounding.

Table 5: Comparison of scores of cohort project and control groups between 2007 and 2010

Mean score (%)Project GroupPre-test (2007) 18.4Post-test 1 (2007) 36.8Post-test 2 (2010) 40.8Change (2007 to 2010) 22.5Control Group Pre-test (2007) 16.4Post-test 1 (2007) 19.0Post-test 2 (2010) 32.9Change (2007 to 2010) 16.5Difference (Impact) +6.0

6. Baseline Data: 2010

These scores will be used to compare performance over the next two years to measure impact of the programme while under departmental management.

The tables that follow report learner scores in terms of percentages – see Appendix Seven for the raw scores from which the percentages are derived.

Scores for Part One in both instruments are of particular interest in that they are the items that were used for LO1 in the NSE instrument.

Table 6: Baseline Scores: Grade 4 cohort (%)n Part One Part Two Part Three Part Four Total

Malamulele Central 758 31.7 35.6 28.8 25.7 31.8Malamulele East 726 35.9 38.2 29.7 33.0 35.2Malamulele North East 859 33.8 37.3 30.3 26.4 33.4Malamulele West 653 33.0 33.7 31.4 27.1 32.0Vhumbedzi 732 31.6 34.2 31.3 30.2 32.4Whole sample 3 728 33.2 35.9 30.3 28.4 33.0

Table 7: Baseline Scores: Grade 6 (%)

n Part One Part Two Part Three Part Four Part Five TotalMalamulele Central 136 26.9 32.4 17.2 26.2 36.4 28.1Malamulele East 150 33.0 36.8 19.8 30.3 36.3 31.3Malamulele North East 227 30.6 40.8 16.2 29.6 42.8 32.4Malamulele West 264 28.7 32.4 17.7 27.3 34.7 28.3Vhumbedzi 329 27.6 33.8 16.2 27.9 36.0 28.4Whole sample 1 106 29.1 35.1 17.2 28.2 37.2 29.5

16

ESA


It is clear that learner performance is still very poor – none of the sub-samples obtained a ‘pass mark’ of at least 50% for any of the parts of the instrument. Nonetheless, performance is generally better than was recorded in the 2007 baseline data, albeit that the 2007 data was obtained from a very much smaller sample in only three of the five circuits from which the 2010 data is derived. The implication is two-fold:

That the Foundations for Learning Campaign has had some effect in these circuits That the PMRP, especially the Mental Mathematics Competition run in support of FFLC has

also had some generic influence on teacher methodology in all of the circuits since 2007.

As might be expected from the mean scores, the scores of individual learners could hardly paint a more depressing reflection of learner performance on a very large scale. The figure below reports the proportions of learners who fell into one of four score ranges.

0% to 24% 25% to 49% 50% to 74% 75% to 100%0

10

20

30

40

50

60

33.5

50.1

14.4

2

39.7

50.6

9.2

0.5

Figure 4 : Proportion of learners in score ranges

Grade 4Grade 6

Score Range

% o

f lea

rner

s

Only 16.4% of Grade 4, and 9.7% of Grade 6, learners achieved a score of 50% or better on an instrument standardized for their respective grade levels. At the very least, this would suggest that the chance of these circuits and learners achieving the targets of the Foundations for Learning Campaign - a mean of 50% or more - are remote indeed!

Part Two of the instrument deals with the basic arithmetic operations and the table below disaggregates the scores into each of these operations.

Table 8: Part Two: Grade 4: Arithmetic operations (%)

Add Subtract Multiply DivideMalamulele Central 32.2 23.7 9.1 8.3Malamulele East 33.0 26.5 10.3 10.1Malamulele North East 33.3 24.9 10.6 9.5Malamulele West 31.6 23.4 10.6 6.2Vhumbedzi 32.4 23.8 10.3 7.4Whole sample 32.6 24.5 10.2 8.4

17

ESA


Table 9: Grade 6 Arithmetic operations (%)

Add Subtract Multiply DivideMalamulele Central 61.5 40.4 10.9 16.8Malamulele East 65.1 49.2 16.0 16.9Malamulele North East 73.0 54.0 17.1 19.2Malamulele West 62.7 43.6 9.2 14.0Vhumbedzi 65.4 45.3 10.5 14.0Whole sample 65.8 46.6 12.3 15.8

As is usual in South African research, mean scores decline as one passes from addition to division. The reason is that learners can use counting of single units (‘sticks’) as against true calculating to arrive at the answer for addition and subtraction but the practice is very much more difficult to control for multiplication and division – it is obvious that learners in none of the circuits at either grade level have any knowledge of the times tables.

None of the indices at Grade 4 level approaches a ‘pass mark’ of 50%, while only in addition at Grade 6 level do all circuits achieve a score of over 50%; in addition, Malamulele North East exceeded this score for subtraction and Malamulele East was very close to it.

In general, it is obvious that very severe problems persist in carrying out fundamental arithmetic operations and, since a facility with basic arithmetic is the foundation of all higher order mathematics, that learners in these circuits cannot be expected to progress in the study of this subject in higher grades.

Eight of these operations (two each) were matched with word sums in which the same arithmetic problem is posed in words.

Table 10: Grade 4: Word sums and matched operations

Word sums Matched operations DifferenceMalamulele Central 25.7 45.8 20.2Malamulele East 33.0 49.9 16.9Malamulele North East 26.4 49.0 22.6Malamulele West 27.1 44.9 17.7Vhumbedzi 30.2 46.2 16.0Whole sample 28.4 47.3 18.9

Table 11: Grade 6: Word sums and matched operations

Word sums Matched operations DifferenceMalamulele Central 26.2 43.7 17.5Malamulele East 30.3 47.3 16.9Malamulele North East 29.6 51.0 21.4Malamulele West 27.3 43.5 16.2Vhumbedzi 27.9 44.0 16.0Whole sample 28.2 45.7 17.5

The disjunction between the word sums and the corresponding operations is immediately apparent. The relationship between language and performance is so familiar in South African education that it

18

ESA


need not be belaboured here. However, while it is obvious that comprehension of the medium of instruction is the precondition for the learning of any subject taught in that language, it is equally obvious that simple comprehension of the language in which it is taught is not a sufficient guarantee of competence in mathematics.

The word sums in the instrument are in very simple English (e.g. Sipho has 5 books and Themba has 4 books. How many books do they have all together?”) and considerable attention was paid to ensuring learners understood the meaning of the questions; even when learners did understand the semantics of the word sums, they had difficulty extracting the operation that was involved.

7. Years Two and Three of the Evaluation

The baseline data will be compared with data derived from the scores obtained by the longitudinal cohort - i.e. Grade 4 in 2010 - in 2011 when they are in Grade 5 and in 2012 when they are in Grade 6.

The baseline scores for Grade 6 reflected in this report will provide a second ‘proxy’ control for the learners in the cohort when they reach Grade 6.

19

ESA

APPENDIX ONE: ESTABLISHING THE FORMAL STATUS OF THE PMRP

Circular: The Limpopo Department of Education and the ‘Back to Basics’ Programme of the Primary Mathematics Research Project in the Malamulele Cluster

Date 10 August 2010From KO Dederen: Senior General Manager: CurriculumTo Vhembe District Senior Manager

Vhembe District GET Mathematics CoordinatorCircuit Managers of:Malamulele CentralMalamulele WestMalamulele North EastMalamulele EastVhumbedzi

This is to confirm that the Primary Mathematics Research Project ‘Back to Basics’ programme, currently operating in all five circuits of the Malamulele Cluster, is a programme of the Limpopo Department of Education operating as a part of the provincial Learner Attainment Strategy of the department.

Following the decision by the Limpopo Department of Education in 2008 to implement the PMRP programme in Vhembe District, the District Senior Manger selected the GET Mathematics Coordinator in the district, Mr S. Mukhacwa, as the LDoE Project Manager. The DSM subsequently set up the Project Steering Committee consisting of the FET, GET and ECD/Foundation-Phase Coordinators, Circuit Managers of the affected circuits and a representative of Principals of the participating schools to provide advice and support services to the programme.

Initially (2009) the programme operated in only three of the five circuits in the Malamulele Cluster. Through the recommendation of the Project Steering Committee, the programme was extended in 2010 to all of the five circuits. The programme was operated in 2009 and the first and second terms of 2010 by Eric Schollar and Associates (ESA) on behalf of the department.

The management of the programme is currently being handed over to the department under the direction of the Project Manager, Mr S. Mukhacwa. The baseline for the evaluation of the programme under departmental management is being operated by ESA. This will involve testing of 55 learners in all 125 schools of the Malamulele Cluster. Further details will be provided shortly but please note that the learner testing will commence on the 30th August, depending on the outcome of the current teacher action, and has the approval and support of the Limpopo Department of Education.

This circular serves to confirm that the Primary Mathematics Research Project, as described above and in other documents, is a programme of the Limpopo Department of Education. Once again, the District Senior Manager is requested to provide the necessary directives on the handover process and the evaluation.

KO DederenSenior General Manager: Curriculum

20

ESA

APPENDIX TWO: STEERING COMMITTEE: AGENDA FOR TRANSFER MEETING AND LIST OF MEMBERS

TRANSFER OF THE MANAGEMENT OF THE PRIMARY MATHEMATICS RESEARCH PROJECT TO THE DEPARTMENT IN THE MALAMULELE CLUSTER

MEETING OF THE PMRP STEERING COMMITTEE WITH ESACOPACOPA LODGE: 5 OCTOBER 2010

AGENDA

1. Arrival 10 am

2. Welcome and opening

3. Circular from SGM: Curriculum re Current status of the PMRP in the Limpopo Department of Education

4. Presentation of report on programme implementation to end of second term submitted to the SGM by ESA

4.1. Questions and discussion of report to SGM

5. Evaluation of the PMRP: 2010 to 2012.

6. Planning for the rest of 20106.1. New workbooks for Grade 4 learners6.2. Appointment, responsibilities and training of Key Teachers, departmental officials

and other members of the PMRP Area Committee to end of year6.3. Programme management and monitoring, and collection of final diagnostic

information by end of year

7. Initial planning for 20117.1. Programme management and monitoring7.2. Diagnostic testing and collection of information7.3. Training of newly appointed mathematics teachers in Grades 4, 5 and 67.4. New workbooks for Grade 4 learners

8. Closure and lunch 1 pm

21

ESA

APPENDIX TWO: STEERING COMMITTEE: AGENDA FOR TRANSFER MEETING AND LIST OF MEMBERS

The members of the Steering Committee are: Mr N. Muloiwa Vhembe District: GET Curriculum Coordinator Mr A. Ravele Vhembe District: Curriculum Advisor Mrs T. Rambau Vhembe District: Foundation Phase Coordinator Mr H. Mabasa Vhembe District: FET Curriculum Coordinator Mr S. Mukhacwa Vhembe District: GET Mathematics Coordinator & PMRP Manager Mr M. Maivha Vhembe District: ABET Mathematics ABET Coordinator & AMESA Mr G. Mahlale Circuit Manager: Malamulele North East Mr S. Malaluke Circuit Manager: Malamulele Central and West (acting) Dr. R. Chabalala Circuit Manager: Malamulele East Mr T. Mathonsi Principal: Mapapila Primary School & AMESA Mr E. Schollar ESA

22

ESA

APPENDIX THREE: NOTES FOR MANAGEMENT AND QUALITY SUPPORT GROUP

ESA

TRANSFER OF THE MANAGEMENT OF THE BACK TO BASICS PROGRAMME OF THE PRIMARY MATHEMATICS RESEARCH PROJECT

TO THE MANAGEMENT OF THE LIMPOPO DEPARTMENT OF EDUCATION

TRAINING NOTES FOR MEMBERS OF THE MANAGEMENT AND QUALITY SUPPORT GROUP OF THE PMRP AREA COMMITTEE

COPA COPA LODGE27 OCTOBER 2010

23

ESA

TRANSFER OF THE PMRP TO THE DEPARTMENT: ROLE OF THE QUALITY MANAGEMENT & SUPPORT GROUP

The PMRP programme in the Malamulele Cluster has been handed over to the management and support of the Limpopo Department of Education. The evaluation that has just started will track the performance of learners over the next three years.

Although the PMRP Project Manager for the department is Mr Mukhacwa, who is also the District GET Mathematics Curriculum Advisor, there is insufficient capacity in the district mathematics curriculum advisory services to provide more than overall management and support to the Malamulele Cluster. Therefore, operational management and quality support to schools will be provided through local-level structures under the management of Mr Mukhacwa. The first is the PMRP Steering Committee which includes the Vhembe District Curriculum Support Services and the five Circuit Managers. The second is the PMRP Area Committee which includes principal/HoD/teacher representatives from the five circuits. The key structure is the Management and Quality Support (MQS) sub-committee of the PMRP Area Committee. This includes two Key Principal/SMT members, and two Key Teachers, from each circuit to total 20 members in all – this means there that there is one member of the sub-committee for every six or seven schools. The MQS members will communicate with, and train, school-clusters in their home circuits, and provide support in other circuits if requested. Principal/SMT members and Key Teachers are trained together, and plan together, to ensure that each knows what the other is doing and can support each other’s efforts. For the same reason, they should also operate together at school-cluster meetings as much as possible. Key Teachers will be able to visit schools in their home circuits to provide support to teachers (especially to train and mentor newly appointed teachers) by arrangement with their Circuit Manager.The MQS should organize at least one support cluster meeting for all of the schools before the end of 2010. This meeting should cover:

The integration of the Back to Basics programme with the rest of the National and Provincial programmes. The PMRP is a programme required by the department. Teachers must use the PMRP books for LO1 but still cover all of the other outcomes using other resources like Foundations for Learning.

The books recently distributed to schools are for this year’s Grade 4 learners – in other words, next year’s Grade 5 learners. Learners at Junior Primary schools should take their workbooks with them to the Higher Primary they attend in 2011.

We are working on book supply for the Grade 4 class in 2011. Until they arrive in 2011, teachers should make photocopies of the appropriate Daily Exercises for learners in Grade 4.

There is a small supply of books for topping up the learner books for Grade 5 this year (i.e. next year’s Grade 6 learners). Give numbers to members of the MQS so that these books can be signed for and supplied to schools.

Learners should complete the Formal Assessment at the end of Week 10 so that the teacher of the next grade knows which learner workbook symbol learners should cover for the different operations in 2011. If teachers cannot administer the Formal Assessment this year, they will have to administer it next year before they can start.

When cluster training will start in 2011. Set a date, time and venue. Establish how many newly appointed teachers, or teachers newly allocated to teach mathematics, and who have not been trained there will be

next year – and at which schools. Establish when the first training will take place. Set a date, time and venue. Ensure that Learner Workbooks are carefully stored over the holidays and are returned to learners next year.

24

ESA

INTEGRATION OF PMRP PROGRAMME WITH OTHER NATIONAL AND PROVINCIAL PROGRAMMES

The PMRP programme (Back to Basics) is the official programme for Learning Outcome One: Numbers, Operations and Relationships in the Malamulele Cluster of circuits. The Back to Basics programme is not an optional or voluntary NGO project. It is an official programme of the Learner Attainment Strategy and is directed by the Senior General Manager: Curriculum. The current objective is to find out whether the Limpopo Department of Education should introduce the programme in all of the circuits in Vhembe and then in all of the Districts in Limpopo.

The NCS and CAPS – due to be implemented in 2011 – are both attempts by the National Department of Education to solve two very serious related problems in South African education. The result of these two problems is that mathematics results in South Africa are amongst the very worst in Africa and in the rest of the world.

Teachers and learners do not follow the same content in each term and year (there is no syllabus of study). Therefore, there is no ‘portability’ in the system – what learners learn and at what level of demand is an accident of which schools they go to or which teachers they have. Therefore when they move from one school to another, or one province to another, or even from one teacher to another in the same school, no one knows what to expect, what they should have learned or what content their assessments refer to.

Because of effective automatic progression (in age cohorts), learners are promoted from one grade to the next irrespective of whether they have grasped the content of the previous grades. Therefore, teachers cannot know what to expect of learners, and are faced in every class with learners from every level of ability from the completely innumerate to the very few who are genuinely competent – all classes have become multi-grade in terms of the ability levels of the learners. It is impossible for teachers to support learners at all of these different levels of ability while always teaching content at the correct Assessment Standards.

To support the effective delivery of the NCS (and, soon, CAPS), the Provincial Department of Education has introduced: Common work schedules to ensure that all learners cover more or less the same content each year at the correct assessment standards for each

grade. Common assessments to ensure that all learners are assessed on more or less the same content, and at the correct assessment standards, for each

grade.

There are three other official support programmes that have been introduced to schools: Foundations for Learning which describes the minimum content to be learned, and the minimum milestones to be achieved at the correct

Assessment Standards, for each grade. It is not a prescriptive or complete syllabus of study that replaces the NCS. Assessment Resource Banks which provide teachers with examples for setting their own internal assessment tasks at the correct Assessment

Standards for each grade. They are not textbooks or a programme of study. Mental Mathematics booklets which are intended to provide learners with regular practice of mental mathematics to increase speed and

accuracy of calculations by learners. They are not textbooks or a programme of study.

25

ESA

INTEGRATION OF PMRP PROGRAMME WITH OTHER NATIONAL AND PROVINCIAL PROGRAMMESNote that the Back to Basics materials conform to, and include, all of these policies and programmes. They:

are based on a paced work programme, with regular assessments, based on the Assessment Standards of the NCS include the minimum content required by the Milestones of Foundations for Learning for LO1 – in fact, they include all of the content required

by the NCS in LO1, especially for operations provide multiple examples of assessment tasks based on Assessment Standards provide daily exercises in Mental Mathematics

The Back to Basics programme is a part of the Learner Attainment Strategy in Limpopo which is aimed at improving the performance of learners to the correct ability levels for their grades. Policy requires teachers to teach and assess learners at the correct Assessment Standards for their grades. But in mathematics learners cannot be expected to jump over missed content and suddenly begin to perform at their expected levels. For example, a learner in Grade 6 who has an actual ability level of Grade 3 will not benefit from teaching pitched at Grade 6 level. This learner needs to master the missed Grade 4 and Grade 5 content before he/she can be expected to understand and benefit from teaching content at Grade 6 level. This explains the reason why the Limpopo Department of Education has introduced the PMRP programme in schools.

The Back to Basics programme of the PMRP is a catch up programme that allows learners to start learning at their own level and then to catch-up on the content they missed in previous grades. Unless learners are able to catch-up content that they have missed there is no chance that they will achieve the performance levels required by National and Provincial policy. Therefore, the Limpopo Department of Education has provided the PMRP programme for LO1 to schools to assist teachers to achieve the objectives of National and Provincial policy.

Teachers should follow the common work schedule of the department and use the Back to Basics PMRP programme for LO1 to do so. They should remember also that most Assessment Standards in other outcomes depend on a LO1 skill and, therefore, the Back to Basics programme should be used throughout the year as a support when dealing with any outcome - to ensure the acquisition by learners of the essential LO1 skills they need before they can be expected to achieve the Assessment Standards required by other outcomes.

When dealing with the rest of the outcomes, teachers have the Foundations for Learning files for support, as well as the text books available to the school.

Teachers should only use the Assessment Resource Banks (ARB) as a source of examples of problems set at different Assessment Standards when setting their own internal assessment tasks for learners. Common Assessments will not, in future, contain questions copied from the ARB.

Teachers should use the Mental Mathematics booklets as additional support to the skills developed by the Back to Basics programme. Perhaps as a source for home work tasks, or use them for class competitions or challenge other classes or schools. They should continue to implement the Quick Test at the start of Days 1 to 4 contained in the Back to Basics Learner Workbooks.

26

ESA

KEY DESIGN FEATURES OF THE PMRP PROGRAMME

It deals with Intermediate Phase because these grade levels represent the crucial developmental link between fundamental numeracy at Foundation Phase and more advanced mathematical education from Senior Phase onwards.

It deals with the fundamental conceptual and procedural competencies of numeracy described in the Assessment Standards (AS) of the National Curriculum Statement (NCS) for Learning Outcome One (LO1): Numbers Operations and Relationships. The acquisition of these competencies is the essential requirement for achievement in the other Learning Outcomes and in higher-order mathematics in general. Learners must move beyond simple counting methods and develop the ability to calculate if they are to be able to deal with problems in any LO. For example, LO4 requires learners to calculate Volume which is defined as l x b x h; this requires the prior ability to multiply which is a LO1 skill.

It deals with the problem of ‘multi-grade’ classes in most South African schools. Local and international research agrees that around 80% of our learners are below the minimum expected competency level for the grade levels in which they are enrolled. Learners have persistently been promoted from grade to grade without mastering the content and skills at each successive level. Teachers at all levels are faced with classes in which learners exhibit an enormous range in ability from the innumerate to the very few who are genuinely mathematically competent. The Learner Workbooks, consequently, provide lessons based on the AS for all levels from Grades 3 to 6.

It is based on a diagnostic and continuous assessment system that is intended to allow learners to work with and progressively acquire the skills and content they have missed in previous grades. Mathematics is an hierarchical subject and Grade Six learners, for example, who have a Grade Two level of competence cannot simply leap over the skills and procedures required by Grades Three, Four and Five and be expected to suddenly achieve a Grade Six level of performance.

It is based on the principles of direct instruction, combined with memorization and extensive regular practice of newly learned content, before extensions into ‘learner-centred’ activities requiring minimal teacher instruction and guidance are attempted. The emphasis is placed on the teacher teaching and the learner learning. Attention is paid to the central importance of memory, and accessing information transferred from short to long term memory for problem-solving, in the learning process.

27

ESA

DIAGNOSTIC AND ASSESSMENT SYSTEM AND USING THE PMRP LEARNER WORKBOOKS

The objective of the assessment system is to ensure that learners are allowed to practice the content taught by the teacher at their own level of understanding and progress to higher levels from that starting point.

Learners enter the programme through a diagnostic test which measures their actual grade-level ability for each of the four operations for the grade levels they have supposedly already passed:

Grade 4 learners are tested to see whether they have grasped the content at Grade 3 level. Grade 5 learners are tested to see whether they have grasped the content at Grade 3 and 4 levels Grade 6 learners are tested to see if they have grasped the content at Grades 3, 4 and 5 levels.

Using the results of the diagnostic test, each learner is allocated to the Learner Workbook group for each operation that corresponds to these actual ability levels:

COW = has not grasped content for Grade 3 level = will work on exercises at Grade 3 levelELEPHANT = has grasped content for Grade 3 level = will work on exercises at Grade 4 levelGOAT = has grasped content at Grade 3 and 4 levels = will work on exercises at Grade 5 levelLION = has grasped content for Grade 3, 4 and 5 levels = will work on exercises at Grade 6 level

At the start of the programme, each learner will probably be at different actual ability levels for each operation. For example, Chauke AK in Grade 6 may be at Lion level for addition, Goat level for subtraction, Elephant level for multiplication and Cow level for division. In other words, Chauke AK only works on the exercises marked by the Lion symbol for addition, exercises marked by the Goat symbol for subtraction, exercises marked by the Elephant symbol for multiplication and exercises marked by the Cow symbol for division. Another Grade 6 learner may, for example, be working on exercises for Goat level for both addition and subtraction, and on exercises at Cow level for both multiplication and division - each learner follows his own programme of exercises and practice.

Learners should not work on exercises for all ability symbols one after the other for each week or operation. For example, if a learner is allocated to Goat ability level for addition, he or she will be asked to complete only the Goat exercises – he/she must not do Cow then Elephant then Goat then Lion. If the same learner is allocated to Elephant ability level for subtraction, he/she must be asked to complete only the Elephant exercises for subtraction - he/she must not do Cow then Elephant then Goat then Lion.

Only once a learner has mastered one level can he/she move onto the exercises for the next symbol – i.e. work at the next grade level.

28

ESA

METHOD FOR INTERPRETING DIAGNOSTIC TESTS AND FORMAL ASSESSMENT:WEEK 10HOW TO ALLOCATE LEARNERS TO WORKBOOK SYMBOLS

GRADE 4Part One

(Add)Part Two (Subtract)

Part Three (Multiply)

Part Four (Divide)

Scores

Learner Workbook Group (Cow or Elephant)Add Subtract Multiply Divide

Learners should be allocated to COW group for any score of 6 or less. Scores of 7 and 8 are allocated to ELEPHANT group

GRADE 5Set

Part Operation One Two Group Allocation

One Addition

Two Subtraction

Three Multiplication

Four Division

Use the Set Scores to allocate learners to a Learner Workbook Group for each Part/Operation .

First look at the score for Set One. If the score for Set One is 6 or lower, the learner should be allocated to Cow group. Only if the score for Set One is 7 or 8, look at the scores for the next Set.

If the score for Set Two is 6 or lower, the learner should be allocated to Elephant Group If the score for Set Two is 7 or 8, the learner should be allocated to Goat Group

29

ESA

METHOD FOR INTERPRETING DIAGNOSTIC TESTS AND FORMAL ASSESSMENT:WEEK 10HOW TO ALLOCATE LEARNERS TO WORKBOOK SYMBOLS

GRADE 6Set

Part Operation One Two Three Group Allocation

One Addition

Two Subtraction

Three Multiplication

Four Division

Use the Set Scores to allocate learners to a Learner Workbook Group for each Part/Operation .

First look at the score for Set One. If the score for Set One is 6 or lower, the learner should be allocated to Cow group. Only if the score for Set One is 7 or 8, look at the scores for the rest of the Sets.

If the score for Set Two is 6 or lower, the learner should be allocated to Elephant Group If the score for Set Two is 7 or 8, but the score for Set Three is 6 or lower, the learner should be allocated to Goat Group If the score for Set Two is 7 or 8 and the score for Set Three is also 7 or 8, the learner should be allocated to Lion Group.

30

ESA

STRUCTURE OF THE PMRP PROGRAMME

YEAR ONEInitial diagnostic test Measure actual ability level for each operation for each learnerStart programme Use results to allocate each learner to ability symbol for each operation. Learners work only at this levelDay 3 and 4 of each week Record marks for learner exercises to assess which learners need help and which can be given extension exercises

on Day 5Day 5 of each week Revision and extension day: Provide re-teach/catch-up and extension exercises for learners, as applicable

End Week 4 Record marks for assessment of addition and subtraction to assess which learners need help and which can be given extension exercises for these operations in Week 10

End Week 9 Record marks for assessment of multiplication and division to assess which learners need help and which can be given extension exercises for these operations in Week 10

Week 10 Revision and extension week: Provide re-teach/catch-up and extension exercises for learners for 4 operations depending on results of Week 4 and Week 9 assessments

Formal assessment at end of Week 10 Measure actual ability level for each operation for each learner for next year

Start Week 11 Learners allocated to lowest ability level (Cow to Lion) obtained for any of the operations for these weeksWeeks 11 to 14 Learners work on work on the exercises marked by the relevant symbol for this ability levelYEAR TWO

Start programme Use results of formal assessment at the end of Week 10 in Year One to allocate each learner to ability symbol for each operation. Learners work only at this level

Initial assessment

through diagnostic test

for the 4 operations

before programme

starts

Allocate to groups in Learner

Workbook (cow,

elephant, goat, lion) for

each operation

Continuous assessment

through exercises on Days 3 and 4 of each Week

and tests at the end of

Weeks 4 & 9

Formal assessment

through diagnostic test

for the 4 operations at

the end of Week 10

Allocate to groups in Learner

Workbook (cow,

elephant, goat, lion) for

each operation for

next year

31

ESA

SIMPLIFIED STRUCTURE OF A PMRP LESSON ON DAYS ONE TO FOUR

Stage and Timing of Lesson

1. Preparation before lesson Before lesson, carefully read through Teacher Manual & Learner Workbook. Write required work indicated in Teacher Manual on board before the lesson starts and before the learners arrive in the class.

2. Mental Arithmetic: ± 10 minutes

Whole process takes no more than 10 minutes from start to finish. Learners are allowed only 3 minutes of this time to complete the test whether they are finished or not. Books are exchanged, teachers read out the answers and learners mark each other’s work. Teachers quickly review performance but do not mark or record the quick tests. These tests are intended to increase the speed and accuracy of learners’ numeracy skills through practice.

3. Teaching: ± 20 minutes

The teacher teaches content for 20 minutes using the preparation written on the board before the lesson. Teachers should teach directly from the Teacher Manual. Teachers should directly teach and directly explain the content of the lesson to learners who listen, watch and learn. It is very important that teachers realize that their role here is as a teacher and not as a ‘facilitator’. Learners should not be asked to solve problems on the board, or in pairs/groups, while you are teaching because only those who already understand the content will be able to do while the others will receive no input – exactly the learners who most need it! Only once the teacher has completed his/her direct teaching of the content should he/she ask a few learners a question or two – concentrating on those learners the teacher knows are weak – to check how well the content was presented.

4. Learner Exercises: ± 30 minutes

Individual learners (pairs of similar ability learners are also usable) complete exercises based on the content they have just learned for 30 minutes. They should be in rows not in groups. Teachers should circulate constantly and interact with individual learners – weak and strong. Teachers now have plenty of time to help gauge how well the class has understood the content teaching presented that day. Do not use this time for marking learners’ work!

32

ESA

EXTENSION, RE-TEACHING, HOMEWORK AND THE USE OF OTHER EXISTING RESOURCES

Teachers should use all of the available existing materials to support the use of the PMRP programme.

An abacus is an excellent means for explaining place value and the base-10 number system.

Old text books are an excellent source of additional exercises for learners. For example, learners who have achieved very good marks on Days Three and Four can be provided with books at one grade level higher and challenged to attempt problems of the same sort at this higher level. Equally, learners who are having particular problems with some element of content can be provided with extra exercises from books at one grade level lower.

Class sets of old pre-OBE books still stored by schools can be used very profitably for homework.

33

ESA

QUICK CHECKLIST FOR PROGRAMME MANAGEMENT

Ensure the diagnostic process is implemented, records kept and learners allocated to workbook symbols.

Ensure teachers understand how to use correctly the assessment system from the initial diagnostic test to the final formal assessment.

Ensure the teachers implement the programme by teaching directly from the Teacher Manual and using the Learner Workbooks. Teachers should write the date when each lesson is presented in both the Teacher Manual and the Learner Workbook.

Ensure teachers complete the whole 14 Week programme, and keep to an appropriate pace while doing so. Days missed cannot be skipped but must be made up before the teacher proceeds to the next Day.

Ensure teachers reach, at the very least, the end of Week 10 so that the learners can again be assessed for the start of the next year.

Ensure that newly appointed teachers are orientated and trained

Ensure that teachers apply the correct Lesson Structure on Days One to Four

Ensure that teachers use each Day 5 for catch-up/re-teaching/revision and for extension, as appropriate

Ensure that records of diagnostic and formal testing are collected as management information

Ensure that Learner Workbooks are carefully cared for and stored after hours, and during holidays. Most learners in Grade 4 are expected to use their books for Grade 5 and 6 as well. This provides learners with a complete record of all the work they did during Intermediate Phase, and helps to restrain the cost of supplying new books to the programme each year.

34

ESA

APPENDIX FOUR: MONITORING INFORMATION BY CIRCUIT AND SCHOOL

Central East North East West Vhumbedzi George Hasani 2 Fumani 1 Botsoleni 3 Chanyela 1 Begwa Shitumbe 2

Khanani 2 Govhu 3 Boxahuku 3 Gingirikani 2 Denzhe 3Khupukani 3 Hangalakani 3 Gonani 3 Guwela 2 Duvhuledza 2Langutani 2 Hisekelani 1 Hlengani 3 Hasani Lawrence 2 Gaba 3Magangeni 2 Hitekani 2 Joas Phahlela 1 Hlawulekani 3 Gonela 2

Magoda 3 Hoji 2 Madzikuse 2 Khakhan'wa 1 Gunda 2Mahonisi 2 Khodobi 1 Magomani 3 Machele 3 Khavuwe 2

Makhapule 1 Mabayeni 2 Mahlohlwani 2 Mahlefunye 2 Lambani 2Makumeke 2 Magona 3 Makahlule 3 Mudabula 2 Lukalo 3Manavele 1 Makhasa 1 Makuleke 2 Mukhomi 3 Mahagala 3Mapapila 1 Manghena 1 Maledza 3 Mulamula 2 Manzemba 3Mavambe 3 Mashobye 1 Maphophe 3 Mulenzhe 2 Masetoni 3Mavuyisi 3 Merwe 2 Mashakadzi 2 Muswane 1 Mbofheni 3

Mdanisi 2Nghomunghom

u 2 Matiyani 3 Mutoti 2 Mubvumoni 2Mutshena 3 Nkandziyi 3 Mayeke 2 Mheho 2 Mushiru 3

Ripindzi 3Nxanguyintshw

a 2 Mhinga 1 Nhomelani 2 Muthuli 3Shibangwa 3 Nyavani 2 Minga Special 1 Phaphazela 1 Mutshetshe 3

Shimambani 2 Phatima 3 Mphakati 3 Risana 2 Nyamuliwani 2Shigalo 3 Shigombe 2 Nkhavi 2 Tlangelani 3 Pfukoni 2

Shitlhelani 1 Shikatsa 2 Phaweni 3 Tshamani 2 Thomani 2Titirheleni 2 Shilume 1 Rhangani 2 Tshamiseka 2 Tshamutshedzi 2Tivanani 3 Tinyiko 1 Shilungwa 2 Tovhowani 1 Tshaulu 2Tiyiselani 3 Tivoneleni 3 Sunduza 2 Twananani 1 Tshidzini 2

Tsundzukani 2 Tshikiwani 2 Tlhelani 1 Tshifudi 3Xihlovo 1 Tshikonelo 1

Vongani 2

35

ESA

APPENDIX FOUR: MONITORING INFORMATION BY CIRCUIT AND SCHOOLXimixoni 2

36

ESA

APPENDIX FIVE: NUMBERS OF SCHOOLS AND LEARNERS TESTED BY CIRCUIT

Malamulele Central Malamulele East Malamulele North East Malamulele West VhumbedziID School n ID School n ID School n ID School n ID School n1 George Hasani 19 1 Fumani 40 1 Botsoleni 29 1 Chanyela 40 1 Begwa-Tshitumbe 402 Khanani 39 2 Govhu 27 2 Boxahuku 40 2 Khakhanwa 46 2 Denzhe 203 Khupukani 35 3 Hangalakani 40 3 Gonani 16 3 Machele 34 3 Duvhuledza 264 Magangeni 29 4 Hisekelani 23 4 Hlawulekani 40 4 Madubula 40 4 Gaba 405 Magoda 15 5 Hitekani 21 5 Hlengani 40 5 Mahlefunye 39 5 Gonela 436 Mahonisi 18 6 Khodobi 22 6 Joas Phahlela 44 6 Mheho 41 6 Gunda 267 Makhapule 26 7 Magona 40 7 Mabayeni 35 7 Mukhomi 40 7 Lukalo 338 Makumeke 27 8 Makhasa 34 8 Madzikuse 22 8 Mulamula 40 8 Mahagala 259 Manavele 40 9 Manghena 31 9 Magomani 40 9 Mulenzhe 40 9 Manzemba 3910 Mapapila 20 10 Mashobye 38 10 Mahlohlwani 18 10 Muswane 40 10 Masetoni 3111 Mavambe 25 11 Merwe 26 11 Makahlule 26 11 Nhombelani 22 11 Mmbofheni 4212 Mavuyisi 33 12 Nghomunghomu 43 12 Makuleke 48 12 Phaphazela 40 12 Mubvumoni 1513 Mdanisi 23 13 Nkandziyi 35 13 Maledza 43 13 Risana 40 13 Mushiru 2814 Mutoti 10 14 Nxanguyintshwa 40 14 Maphophe 30 14 Tlangelani 40 14 Muthuli 4415 Mutshena 11 15 Nyavani 15 15 Mayeke 40 15 Tovhowani 30 15 Mutshetshe 4116 Phaweni 46 16 Phatima 40 16 Mphakhathi 40 16 Tshamani 41 16 Nyamuliwani 4017 Ripindzi 30 17 Shigombe 40 17 Nkhavi 32 17 Tshamiseka 40 17 Pfukoni 1418 Shibangwa 22 18 Shikatsa 36 18 Rhangani 40 18 Thomani 4119 Shigalo 41 19 Shilume 25 19 Shilungwa 40 19 Tshamutshedzi 3320 Shimambani 18 20 Tinyiko 90 20 Sunduza 41 20 Tshaulu 3421 Shitlhelani 40 21 Tivoneleni 20 21 Tlhelani 41 21 Tshidizini 3722 Titirheleni 28 22 Tshikonelo 40 22 Tshifudi 4023 Tivanani 40 23 Vongani 3924 Tiyiselani 41 24 Ximixoni 3525 Tshikiwani 2326 Tsundzukani 4027 Xihlovo 19

Total 758 Total 726 Total 859 Total 653 Total 732

37

ESA


Grade 6

Malamulele Central Malamulele East Malamulele North East Malamulele West VhumbedziID School n ID school n ID school n ID school n ID school n

1 Khupukani 15 1 Fumani 15 1 Hlawulekani 15 1 Chanyela 15 1 Denzhe 132 Langutani 15 2 Hoji 15 2 Hlengani 15 2 Gingirikani 15 2 Duvhuledza 153 Makumeke 15 3 Mabayeni 15 3 Joas Phahlela 15 3 Guwela 30 3 Gaba 154 Manavele 15 4 Merwe 15 4 Madzikuse 15 4 Hasani Lawrence 16 4 Gonela 165 Mutoti 19 5 Phatima 15 5 Makuleke 15 5 Khakhanwa 15 5 Gunda 15

6 Mutshena 12 6Shigombe 15 6 Maledza 15 6 Machele 15 6 Khavuwe 16

7 Phaweni 15 7 Shikatsa 15 7 Mashakadzi 16 7 Mahlefunye 15 7 Lukalo 158 Tshikiwani 15 8 Shilume 15 8 Mayeke 15 8 Mheho 15 8 Mahagala 15

9Tsundzukani 15 9 Tinyiko 30 9 Mhinga 15 9 Mudabula 15 9 Manzemba 15

10 Mphakhathi 15 10 Mulamula 15 10 Masetoni 1611 Rhangani 15 11 Mulenzhe 15 11 Mmbofheni 1512 Tlhelani 15 12 Muswane 15 12 Mubvumoni 1513 Tshikonelo 15 13 Nhombelani 15 13 Mushiru 1414 Vongani 16 14 Risana 15 14 Muthuli 1515 Ximixoni 15 15 Tovhowani 8 15 Mutshetshe 15

16 Tshamiseka 15 16 Nyamuliwani 1517 Twananani 15 17 Pfukoni 12

18 Thomani 1619 Tshamutshedzi 1520 Tshaulu 1621 Tshidzini 1522 Tshifudi 15

Total 136 Total 150 Total 227 Total 264 Total 329

38

ESA


2007 Cohort

Malamulele North East Malamulele East Malamulele CentralProject n Control n Total Project n Control n Total Project n Control n TotalBoxahuku 18 Botsoleni 33 Hangalakani 32 Govhu 40 George Hasani 36 Khanani 33

Gonani 8Mahlohlwani 35 Hisekelani 34 Makhasa 31 Magangeni 26 Mahonisi 35

Magomani 23 Matiyani 37 Khodobi 19 Manghena 35 Magoda 48 Mavuyisi 20

Makahlule 29Nghomunghomu 41 Mashobye 30 Makhapule 37 Mdanisi 35

Maphophe 38 Nyavani 45 Nkandziyi 20 Mapapile 39 Shimambani 41Nxanguyintswha 27 Mavambe 48 Tivanani 32Tivoneleni 27 Ripindzi 45 Tiyiselani 36

Shigalo 26 Xibangwa 27Titirheleni 44Xihlovu 36

Total 116 105 221 Total 171 Total 210 381 Total 385 Total 259 644

39

ESA

APPENDIX SIX: ITEM ANALYSIS OF TEST INSTRUMENTS

Grade 4

Difficulty index Count % Difficulty index Suggested % Actual %

90 – 100 11 22% > 75 25 55%80 – 89 8 16% 25 - 75 50 33%70 – 79 9 18% < 25 25 12%60 – 69 4 8% 100 10050 – 59 8 16%

40 – 49 1 2%

30 – 39 2 4%

20 – 29 4 8% Discrimination index Count %

10 – 19 1 2% 40+ 41 84%0 – 9 1 2% <40 8 16%

Total number of items 49 100% Total number of items 49 100%

40

ESA

APPENDIX SIX: ITEM ANALYSIS OF TEST INSTRUMENTS

Grade 6

Difficulty index Count % Difficulty index Suggested % Actual %

90 – 100 21 25% > 75 25 48%80 – 89 13 15% 25 - 75 50 49%70 – 79 12 14% < 25 25 4%60 – 69 16 19% 100 100%50 – 59 7 8%

40 – 49 6 7%

30 – 39 4 5%

20 – 29 3 4% Discrimination index Count %

10 – 19 2 2% 40+ 52 62%0 – 9 0 0% <40 32 38%

Total number of items 84 100% Total number of items 84 100%

41

ESA

APPENDIX SEVEN: RAW SCORES AND PERCENTAGES BY CIRCUIT

Table 12: Grade 4: Scores by Parts of the Test and Total

Part One (12 items) Part Two (20 items) Part Three (9 items) Part Four: 8 (items) Total: (49 items)n Score % Score % Score % Score % Score %

Malamulele Central 758 3.8 31.7 7.1 35.6 2.6 28.8 2.1 25.7 15.6 31.8Malamulele East 726 4.3 35.9 7.6 38.2 2.7 29.7 2.6 33.0 17.3 35.2Malamulele North East 859 4.1 33.8 7.5 37.3 2.7 30.3 2.1 26.4 16.4 33.4Malamulele West 653 4.0 33.0 6.7 33.7 2.8 31.4 2.2 27.1 15.7 32.0Vhumbedzi 732 3.8 31.6 6.8 34.2 2.8 31.3 2.4 30.2 15.9 32.4Whole sample 3 728 4.0 33.2 7.2 35.9 2.7 30.3 2.3 28.4 16.2 33.0

Table 13: Grade 6: Scores by Parts of the Test and Total

Part One (16 items) Part Two (20 items) Part Three (20 items) Part Four (8items) Part Five (20 items) Total (84 items)n Score % Score % Score % Score % Score % Score %

M. Central 136 4.3 26.9 6.5 32.4 3.4 17.2 2.1 26.2 7.3 36.4 23.6 28.1M. East 150 5.3 33.0 7.4 36.8 4.0 19.8 2.4 30.3 7.3 36.3 26.3 31.3M. North East 227 4.9 30.6 8.2 40.8 3.2 16.2 2.4 29.6 8.6 42.8 27.2 32.4M. West 264 4.6 28.7 6.5 32.4 3.5 17.7 2.2 27.3 6.9 34.7 23.7 28.3Vhumbedzi 329 4.4 27.6 6.8 33.8 3.2 16.2 2.2 27.9 7.2 36.0 23.8 28.4Whole sample 1 106 4.7 29.1 7.0 35.1 3.4 17.2 2.3 28.2 7.4 37.2 24.8 29.5

42

ESA


Table 14: Part Two: Grade 4: Arithmetic operations (8 items each)

Add Subtract Multiply DivideScore % Score % Score % Score %

Malamulele Central 1.6 32.2 1.2 23.7 0.5 9.1 0.4 8.3Malamulele East 1.7 33.0 1.3 26.5 0.5 10.3 0.5 10.1Malamulele North East 1.7 33.3 1.2 24.9 0.5 10.6 0.5 9.5Malamulele West 1.6 31.6 1.2 23.4 0.5 10.6 0.3 6.2Vhumbedzi 1.6 32.4 1.2 23.8 0.5 10.3 0.4 7.4Whole sample 1.6 32.6 1.2 24.5 0.5 10.2 0.4 8.4

Table 15: Grade 4: Word sums and matched operations (8 items each)

Word sums Matched operations DifferenceScore % Score % Score %

Malamulele Central 2.1 25.7 3.7 45.8 1.6 20.2Malamulele East 2.6 33.0 4.0 49.9 1.4 16.9Malamulele North East 2.1 26.4 3.9 49.0 1.8 22.6Malamulele West 2.2 27.1 3.6 44.9 1.4 17.7Vhumbedzi 2.4 30.2 3.7 46.2 1.3 16.0Whole sample 2.3 28.4 3.8 47.3 1.5 18.9

43

ESA


Table 16: Part Two: Grade 6: Arithmetic operations (8 items each)

Add Subtract Multiply DivideScore % Score % Score % Score %

Malamulele Central 3.1 61.5 2.0 40.4 0.5 10.9 0.8 16.8Malamulele East 3.3 65.1 2.5 49.2 0.8 16.0 0.8 16.9Malamulele North East 3.7 73.0 2.7 54.0 0.9 17.1 1.0 19.2Malamulele West 3.1 62.7 2.2 43.6 0.5 9.2 0.7 14.0Vhumbedzi 3.3 65.4 2.3 45.3 0.5 10.5 0.7 14.0Whole sample 3.3 65.8 2.3 46.6 0.6 12.3 0.8 15.8

Table 17: Grade 6: Word sums and matched operations (8 items each)

Word sums Matched operations DifferenceScore % Score Score % Score

Malamulele Central 2.1 26.2 3.5 43.7 1.4 17.5Malamulele East 2.4 30.3 3.8 47.3 1.4 16.9Malamulele North East 2.4 29.6 4.1 51.0 1.7 21.4Malamulele West 2.2 27.3 3.5 43.5 1.3 16.2Vhumbedzi 2.2 27.9 3.5 44.0 1.3 16.0Whole sample 2.3 28.2 3.7 45.7 1.4 17.5

44

ESA

Documents

30 Nymphe Street - The Evaluation Research Agency 2011_PMRP … · Web viewThe word sums in the instrument are in very simple English (e.g. Sipho has 5 books and Themba has 4 books