Evaluation Design of the Cofimvaba e-Textbook Initiative

Benita Williams, Mario Marais, Mmamakanye Rampa

Project Aims

• The project aims to improve the teaching and learning environment

• Contribute towards – improved learner results and – the development of 21st century skills among

teachers and learners.

• It also aims to document a model for e-textbook rollout in schools, which could potentially be sustainably replicated in other educational districts.

12 Component Implementation Model

Merryl Ford, CSIR Meraka Institute

SCHOOL ICT ARCHITECTURE

DevicesWireless LAN

Storage and Power

CHANGE MANAGEMENTPeople (District, SMT)

TechnologyProcess

CONTENTStandardsConversion

Creation & Customisation

TEACHER DEVELOPMENTTraining

PreparationIn the classroom

OPERATIONS MANAGEMENT

Logistics Support & Maintenance

Distribution

RESEARCHMasters & PhD Technology R&D

ICT4E R&D

NETWORKWiFi Mesh

Backbone connectivityInternet

MONITORING & EVALUATION

LearnersTeachersSchool

COMMUNICATIONMarketing strategy

Social Media StrategyKnowledge management

COMMUNITY ENGAGEMENT

Learners & ParentsTeachers

Community

STAKEHOLDER MANAGEMENT

District/Circuit officialsLocal leadership

Provincial

PROJECT MANAGEMENTFinancial Management

ProcurementImplementation management

M&E Approach

• Developmental Evaluation• Principles of Utilization Focused Evaluation

• Why? Constantly evolving complex and innovative project

Framework Part 1

• Key Activities– Model Development

• Document Theory of Action & Theory of Change• Design Choice Review• Cost Study

– Monitoring & Learning• Implementation Monitoring

– Evaluation & Learning• Result Review (Reaction, Learning, Individual Behaviour,

Organizational change at level of learner, teacher, school, district, province)

Framework Part 2Component Evaluation QuestionsDocumentation of theory of action and theory of change

1. What is the initial design and significant modifications made to each of the 12 components (version 1 of model) required to address in e-textbook rollout?

Method: Participatory Workshop

Framework Part 2Component Evaluation Questions

Design choice review

Programme focus: 2. What were the significant design choices for each of the components, and what was the impact on relevance, sustainability and achievement of outcomes?Model focus: 3. Which design of the programme components, and programme overall, are likely to yield sustainable implementation by the DBE/ ECDOE ?

Methods: Documentation of Decisions in a decision template. Documentation of lessons in “Learning brief” template

Framework Part 2Component Evaluation Questions

Cost Study Programme focus: 4. What was the total cost of implementing the different designs of the programme ?Model focus: 5. What would the total cost be to implement such a programme on district wide level? (three scenarios, different designs, in comparison to existing LTSM delivery process)

Methods: GeSCI TCO toolProcess modellingScenario BuildingActivity based costing

Framework Part 2Component Evaluation Questions

Implementation monitoring

Programme focus:6. Was the programme components, and programme overall, feasible to implement?Model focus:7. What have we learnt about the implementation of this programme that should be transferred to others?8. Which factors support, inhibit or prevent implementation?

Methods: Textual analysis of Twitter feed, and Whatsap groupPost training trainer feedback formAttendance registersPost-training teacher feedback forms

Quarterly telephonic teacher surveyQuarterly telephonic school surveyDevice usage statisticsSchool functionality checklist

Framework Part 2Component Evaluation Questions

Results Review: Programme focus: 9. Did the programme deliver the anticipated educational outcomes (learning, behaviour change, organizational change) and necessary intermediate outcomes (reaction) at learner, teacher, school, district and province levels?10. What were the unanticipated results?Model focus: 11. What have we learnt about the results of this programme that should be transferred to others?12. How do the achieved outcomes in this project compare to the outcomes 13. Which factors support, inhibit or prevent the realisation of outcomes?

Framework Part 2• Post-training teacher feedback forms• Learner focus groups, Parent focus groups• Quarterly telephonic district interviews

• Teacher session assessment record (Badge tracker)• Teacher baseline questionnaire and follow up

• Learner 21st century assessment assignment with sample of learners• Assessment of learner marks (Matric results, ANAs) in comparison with other

matched schools • Ethnographic description of classroom observations • School Functionality checklist • Impact Story template• School leavers tracer study

Issues• Initiative is evolving, the M&E framework needs to be

adapted within budgetary and practical constraints • Operationalizing the evaluation required that other

component managers must help with the collection of data, interpretation of the data

• Evaluation information collected on some tools may have immediate uses for other components – how best to make it available

• Capturing learning in learning templates is useful for the team members who are more practice than research oriented.

• Challenges of collecting and packaging the information for purposes of research, evaluation, and sharing in media

Benefits

• The benefit of the evaluation approach include providing continuous feedback on various project components to the various evaluation users whilst also collecting information that could later be consolidated into a model.

• There is flexibility to incorporate methods with different ontological foundations e.g. RCT and thick ethnographic description in order to meet the needs to the diverse set of users.

• The framework is now also being adapted for the evaluation of other similar ICT programmes rolled out by the Meraka Institute