ICT for education: a conceptual framework for the sustainable adoption of technology-enhanced learning environments in schools

This article was downloaded by: [University of Birmingham]On: 03 October 2013, At: 13:56Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Technology, Pedagogy and EducationPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/rtpe20

ICT for education: a conceptualframework for the sustainable adoptionof technology-enhanced learningenvironments in schoolsPatricio Rodríguez a , Miguel Nussbaum b & Lioubov Dombrovskaiac

a Center for Advanced Research in Education, Universidad deChile, Santiago, Chileb Department of Computer Science, Pontificia UniversidadCatólica de Chile, Santiago, Chilec Informatics Department, Universidad Técnica Federico SantaMaría, Santiago, ChilePublished online: 28 Sep 2012.

To cite this article: Patricio Rodríguez , Miguel Nussbaum & Lioubov Dombrovskaia (2012) ICTfor education: a conceptual framework for the sustainable adoption of technology-enhancedlearning environments in schools, Technology, Pedagogy and Education, 21:3, 291-315, DOI:10.1080/1475939X.2012.720415

To link to this article: http://dx.doi.org/10.1080/1475939X.2012.720415

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,

http://www.tandfonline.com/loi/rtpe20

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/1475939X.2012.720415

http://dx.doi.org/10.1080/1475939X.2012.720415

systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

ICT for education: a conceptual framework for the sustainableadoption of technology-enhanced learning environments in schools

Patricio Rodrígueza*, Miguel Nussbaumb and Lioubov Dombrovskaiac

aCenter for Advanced Research in Education, Universidad de Chile, Santiago, Chile;bDepartment of Computer Science, Pontificia Universidad Católica de Chile, Santiago,Chile; cInformatics Department, Universidad Técnica Federico Santa María, Santiago, Chile

(Received 26 October 2011; final version received 23 March 2012)

Currently, the use of information and communication technologies (ICT) ineducation does not conclusively demonstrate significant effects on learning.However, not all ICT usage models are designed to affect student outcomes.Therefore, to accurately study the impact of ICT, the concept of an educationalprogramme supported by ICT must first be defined. The authors propose theICT for Education (ICT4E) programme, an evidence-based framework to deter-mine a model’s ability to produce improvements before having to evaluate itsresults. The framework has four components: implementation, intervention,transference and total cost. Based on an explicit definition of the outcomes pur-sued by a given programme, this framework supports the design of its activitiesto promote sustainable changes in pedagogical practices in schools and calculatethe programme’s total cost. The authors illustrate this with a programme imple-mented in six countries. In Chile, the ICT4E programme forms part of the pol-icy for educational technology, evaluating the effectiveness and scalability ofsuch initiatives.

Keywords: technology-enhanced learning; pedagogic model; ICT effects; scala-bility; sustainability; ICT integration

1. Introduction

Attempts to improve education with modern machines – such as the phonograph,lantern slides and television – date from the early twentieth century (Castro, 2004;Cuban, 1986; Earle, 2002; Reiser, 2001; Snider, 1992). For the most part suchinventions come and go, yet what happens in the classroom looks pretty much thesame (Callister, 1992). Information and communication technologies (ICT) werealso introduced in schools to transform teaching and learning processes and toimprove strategies for better educational attainment (Culp, Honey, & Mandinach,2003; Kozma, 2003; Sunkel, 2006).

Nonetheless, such desired effects are far from becoming a reality (Moonen,2008; Reynolds, Treharne, & Tripp, 2003; Robertson, 2002; Tondeur, van Braak, &Valcke, 2007; Trucano, 2005). A wide range of ICT applications and uses wereanalysed in several impact assessments: programme evaluations, specific educationalsoftware in particular subjects at large scale, research and case reviews, and

*Corresponding author. Email: [email protected]

Technology, Pedagogy and EducationVol. 21, No. 3, October 2012, 291–315

ISSN 1475-939X print/ISSN 1747-5139 online� 2012 Association for Information Technology in Teacher Educationhttp://dx.doi.org/10.1080/1475939X.2012.720415http://www.tandfonline.com

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

meta-analysis research (see further details in section 2). Some of these showed apositive impact, while others a neutral or negative impact, leading many authors toargue that there is still no conclusive answer to the question about the impact andeffectiveness of ICT in education (Burns & Ungerleider, 2003; Cartwright & Ham-mond, 2007; Cox & Marshall, 2007; Kikis, Scheuermann, & Villalba, 2009;Reeves, 2008; Trucano, 2005). Other authors even argue that the introduction ofICT in schools served to amplify existing forms of inequality (Toyama, 2010, 2011;Warschauer, Knobel, & Stone, 2004).

However, ICT was not devised with education in mind, it did not appear naturallyin educational systems, nor was it demanded by the teaching community (Sunkel,2006). It responds to a model – with its concepts, discourse and practices – whichoriginated outside of educational systems (Kikis et al., 2009; Sunkel, 2006) and wasthen imposed upon them (Hennessy, Ruthven, & Brindley, 2005), making theintegration a highly difficult process. This later worsened when greater emphasis wasplaced on technology than on pedagogy (Trucano, 2005; Watson, 2001), ignoring theextensive existing research on teaching and learning (Sutherland et al., 2004).

1.1. Problem definition

When ICT were first being introduced in schools, they were considered so ‘new’that some policy makers and practitioners ignored theoretical foundations central toall learning with or without ICT (Sutherland et al., 2004). In addition to learningtheories, other design aspects of curriculum integration were not considered, suchas the type of ICT resources and how they are specifically used in a particular sub-ject (Cox & Marshall, 2007; McFarlane, 2001) and the design and curriculumimplementation of ICT use in the classroom (Cox & Abbot, 2004; Cox & Marshall,2007; McFarlane, 2001).

By studying ICT integration in schools, it was determined that the way in whichthis takes place depends on the educational context. As part of this context, thefollowing factors were also identified: pedagogical approaches and beliefs (Coburn,2003; Cox & Webb, 2004; Kirkman, 2000; Trucano, 2005), teacher confidence, atti-tudes and skills relating to ICT; school ICT infrastructure, supervision and technicalsupport, involvement and leadership of school principals and time spent by teacherson meetings, training, exercises and lesson planning (Baskin & Williams, 2006;Conlon & Simpson, 2003; Hayes, 2007; Pelgrum, 2001; Reynolds et al., 2003;Tondeur et al., 2007).

Thus, after 30 years of research, the main lesson to have been learned is thatintegrating ICT in education is not a technical matter (Earle, 2002; Trucano, 2005;Watson, 2001). ICT should be the tool with which we deliver content and imple-ment educational practices in better ways, based on curriculum and learning princi-ples (Lai, 2008; Marshall & Cox, 2008; Reeves, 2008; Roblyer, 2005; Trucano,2005). Integration is determined not by the quantity or type of ICT used, but byhow and why it is used (Cox & Abbot, 2004; Cox & Webb, 2004; Earle, 2002;Trucano, 2005). Hereafter, when ICT is used to support teaching and learning activ-ities, we refer to it as technology-enhanced instructional design or technology-enhanced learning (TEL) (Dillenbourg, 2008; Kozma, 1991; Roblyer, 2005).

The available evidence suggests that the use of TEL with specific educationalobjectives, together with an activity that is planned, guided and structured by amediator, is what makes the difference in terms of learning results (Cox & Abbot,

292 P. Rodríguez et al.

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

2004; Pilkington, 2008; Trucano, 2005). However, even with effective interventionsbased on TEL, one problem yet to be resolved is their scalability in the educationsystem. This issue is not exclusive to ICT interventions, and the few existing mod-els of scalability identify four dimensions to consider: depth, sustainability, spreadand shift in reform ownership (Coburn, 2003; Dede, 2006; McDonald, Keesler,Kauffman, & Schneider, 2006). Depth refers to the changes needed in classroompractice, while sustainability refers to how to maintain these changes over time andunder what conditions (Batchelor & Norrish, 2005). Spread (or scalability) refers tothe diffusion of the innovation to large numbers of classrooms and schools, i.e., theconditions needed to replicate or scale up the project, and the impact of doing so(Batchelor & Norrish, 2005). The shift in reform ownership refers to the school’sadoption of the programme.

Since ICT is just one of many available options for supporting teaching andlearning processes (Earle, 2002), it should have to compete for funding with otheralternatives, in terms of cost-effectiveness or cost–benefit, regardless of whetherthese others are based on ICT (Harris, 2009; Margolis, Nussbaum, Rodríguez, &Rosas, 2006). To calculate the total cost, an exhaustive inventory must be made ofthe personnel, software, hardware, materials and resources which are needed toimplement a programme. However, such cost parameters are not set out clearly,as is shown by several studies analysing the cost–benefit of TEL environments(Bates, 2005; Laurillard, 2007; Perraton, Creed, & Robinson, 2002; Rumble, 2001;Steinberg, 2004; Twigg, 2002). Consequently, the cost parameters to be includedand quantified should be specified in operational terms, as well creating a commonlanguage for the educational community (Moyle, 2008).

Finally, in terms of effectiveness, the study of the impact of ICT includes a widerange of technologies and models whose use is not mainly intended to affectstudent attainment. Therefore, before subsequently evaluating their effects, weshould first identify which models produce effective instructional practices. Further-more, any such definition must incorporate the topics of ICT integration in schools,scalability and the total cost of ownership, so that we may establish methods forthe systematic design, implementation and evaluation of said programmes.

1.2. Purpose and structure of this article

In this article, we use the term ‘educational programme’ as a specific set of replica-ble procedures, materials and professional development, whose implementationimproves student attainment and that are based on particular resources and/orinstructional processes (Slavin, 2008). Using this concept, we propose a framework– the ICT for Education (ICT4E) programme – to outline what we understand byan educational programme based on TEL. The objectives of this framework are:

(1) To define the elements, concepts, definitions and critical variables involvedin developing a deep understanding of how and why TEL works in realeducational settings.

(2) To establish the relationships between the expected learning results, theteaching and learning practices supported by TEL, the skills that theserequire, and the activities necessary to develop such skills. This willfacilitate the specification of input, process and outcome indicators for themonitoring and evaluation scheme which must be an integral part of the

Technology, Pedagogy and Education 293

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

programme (Batchelor & Norrish, 2005; Rodríguez, Nussbaum, López, &Sepúlveda, 2010; Wagner et al., 2005).

(3) To define which elements must be present in an ICT4E programme in terms ofthe processes and resources needed to achieve the desired learning results.

(4) To calculate the real total cost of an ICT4E programme, allowing it to becompared with other educational programmes in terms of cost-effectiveness.

Taking the ICT4E programme framework as a base, it is then possible to elabo-rate on the design, implementation and evaluation of its impact.

The rest of this article is structured as follows: in section 2, we present a frame-work for an ICT4E programme based on the evidence available in the literature,indicating its main components, their interrelation and a brief discussion about thedefinition provided. In section 3, we exemplify the framework, using a specific pro-gramme based on computer-supported collaborative learning. In section 4, we dis-cuss the applications of the ICT4E programme that were implemented in Chile. Thearticle finishes with the conclusions and future work in section 5.

2. Defining an ICT for education programme

To outline our framework, we undertook a wide review of the literature on ICT ineducation in three categories: curriculum integration, frameworks for the integrationof ICT in schools and impact assessment.

In the first category – curriculum integration of ICT – we found different frame-works that conceptualise the integration of TEL in teaching/learning models (e.g.contents, activities, roles of the teacher and students). For example, curriculum com-ponents (Van den Akker, 2003), domains of educational innovation (Mioduser,Nachmias, Tubin, & Forkosh-Baruch, 2003), learner-centric ecology of resources(Luckin, 2008), and social infrastructure (Bielaczyc, 2006).

The second category relates to frameworks for the integration of ICT in schools.Here there is an analysis of the factors which affect the implementation of ICT ineducational settings in countries such as Australia (Baskin & Williams, 2006; Hayes,2007), Belgium (Tondeur, Valcke, & van Braak, 2008), Netherlands (Mooij & Sme-ets, 2001), Israel (Nachmias, Mioduser, Cohen, Tubin, & Forkosh-Baruch, 2004;Tubin, Mioduser, Nachmias, & Forkosh-Baruch, 2003), Malaysia (Cloke & Sharif,2001), Great Britain (Kirkman, 2000; Tearle, 2003, 2004) and Singapore (Lim,2002; Lim & Hang, 2003). There are also international studies such as SITESM2(Kozma, 2003; Pelgrum, 2001) and SITES2006 (Plomp, Pelgrum, & Law, 2007).

The third category corresponds to impact assessments of ICT on education,be this in terms of a state of the art summary of the effect of ICT (Burns & Ungerle-ider, 2003; Cox, 2008; Cox & Abbot, 2004; Cox & Marshall, 2007; Cox & Webb,2004; Lai, 2008; Trucano, 2005), considerations for undertaking monitoring andimpact assessments (Agodini, Dynarski, Honey, & Levin, 2003; Light, 2008; Penuel,2005; Wagner et al., 2005), reviews of programmes and cases (Bakia, Means, Galla-gher, Chen, & Jones, 2009; Balanskat, Blamire, & Kefala, 2006; Culp et al., 2003),meta-analysis research (Kulik, 2003; Liao & Hao, 2008; Waxman, Lin, & Michko,2003), studies based on large-scale research design (Barrera-Osorio & Linden, 2009;Campuzano, Dynarski, Agodini, & Rall, 2009; Dynarski et al., 2007) and researchmethodologies about this topic (Batchelor & Norrish, 2005; Cox, 2008; Marshall &Cox, 2008; Pilkington, 2008; Reeves, 2008).


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

Through the second category, we identified three main settings which can beclassified by the aspects of integration of ICT in the school system: classroom,school and external. By ‘classroom’, we mean the setting where teaching and/orlearning occurs in the school, even though it might not take place specificallywithin a classroom (e.g., in a computer lab). ‘School’ defines the specific educa-tional context in which TEL is integrated, while the ‘external’ setting considersthose processes which are outside of school (e.g. monitoring and evaluation, publicpolicies, costing parameters).

The classification of the aspects according to the previously defined settings isshown in Table 1. In this table we include the aspects described in the literaturethat, in our opinion, are the most relevant, some of which are transversal across thedifferent settings (A5 to A10).

Thus, we will define an ICT4E programme as an educational programme whichintegrates the use of a specific TEL environment into teaching and learning prac-tices. Such TEL environments should modify teaching and learning process in waysthat could not be achieved without ICT support. We will call the model for the useand integration of TEL in the teaching/learning process the pedagogic model.

We specify an ICT4E programme by identifying the processes and the relation-ship between them in the integration of the pedagogic model in a school. We assignfour attributes to each process: setting, time, aim and outcomes. Setting refers towhere the process operates (i.e., classroom, school or external), while time refers toits duration. The third attribute is the main aim of the process, which can be classi-fied as one of the following:

(1) Implementation: the use of the pedagogic model by the school team.(2) Intervention: the development of skills and practices in school staff. This is

normally done by an external team which carries out training, ensuring thegradual adoption of the pedagogic model and accuracy of its implementa-tion, using a monitoring and evaluation scheme.

(3) Transference: the professional development of the external team so that theintervention can be accurately carried out on a massive scale.

The fourth attribute is the outcomes, which are the expected skills and/or prac-tices to be developed in the educational actors (i.e., students, teachers, ICT andschool staff). Through the four attributes of the process (setting, time, aim and out-comes), a causal relationship between what happens in the different settings mustbe defined. Therefore, in each process – starting with a set of pre-existing skills andpractices amongst the educational actors – new skills or practices are developed asoutcomes. For example, in the implementation, the outcomes are the learning resultswhich the ICT4E programme promotes amongst the students. In the intervention,the outcomes are the skills and practices which the teachers need in order to per-form implementation. Finally, in the transference, the outcomes are expressed interms of the skills and practices that the external team requires to carry out aneffective intervention.

In sections 2.1 to 2.4 we will describe the implementation, intervention, transfer-ence and calculation of total costs, and how these are grounded in the aspects of theliterature identified in Table 1, to which we will refer with the respective numeration.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

Table

1.Classificatio

nof

theaspectsinvolved

inICTin

educationbasedon

theliterature,

accordingto

thesetting

inwhich

they

occur.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

2.1. Implementation

This is a set of methodological strategies supported by a TEL environment, whoseadoption within a school implies a change or broadening of the existing practices(A5) and the development of new skills amongst the educational actors, with spe-cific roles defined by these strategies (A6). The strategies must consider explicitimprovement goals (A1), defined within an educational context where they operate(A12), based on teaching and learning theories and models (A2) and using specificresources (A3) in a determined configuration (A4). The pedagogic model supportedby TEL is defined by five elements:

• Purpose (A1): particular subjects, levels and specific curriculum contents.• Target schools (A12): schools on which the programme is focused, expressedin terms of geographic surroundings (urban or rural), social vulnerability ofits students and their level of attainment (e.g. measured by local or nationalstandardised tests). Other defining factors could be the type and number ofstudents and school organisation (e.g. multigrade classrooms).

• Implementation resources (A3): all the resources and supporting materialsused by the educational actors (e.g. software, hardware, physical infrastruc-ture, digital and/or non-digital materials, class scripts for teachers and studyguides for students).

• Pedagogic model (A2 to A6): describes how the TEL environment is inte-grated and supports teaching and learning processes. Its main elements are:

○ How they learn: how teaching-learning process are modified (e.g. prepara-tion of a class and its development); how the relationships between actors(e.g. teacher–student, student–student) inside and outside the classroom areaffected; with which configuration (e.g. individual, group) they learn; andhow implementation resources are integrated into existing ones.

○ Assessment of learning results: how expected effects are evaluated.○ Who participates within or outside the school (e.g. teachers, classmates,

parents and family), their roles, and the skills and practices which theyneed to develop.

○ Specific didactic context(s): where (e.g., classroom, computer laboratory orhome) and with whom (i.e., classmates, parents, online tutors) they are used.

These elements can be described using the terminology of existing integrationframeworks, such as curriculum components (Van den Akker, 2003), domainsof educational innovation (Mioduser et al., 2003), learner-centric ecology ofresources (Luckin, 2008), and social infrastructure (Bielaczyc, 2006).

• Implementation outcomes (A1): the expected effects expressed in terms suchas learning results (e.g., scores in standardised tests, learning of specificcurricular contents, acquisition of twenty-first-century skills), personalisedlearning environments, or data collection for monitoring student progress.

2.2. Intervention

So that teachers and students develop the necessary autonomy for sustainable use ofthe pedagogic model (A14) and according to the educational context where they


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

take place (A12), the external team will carry out professional development activi-ties (e.g. teacher training, practical experiences, classroom observation) according toa work plan. The outcomes of such activities are the skills and practices needed bythe different educational actors (A8, A11), which are specific to the proposed peda-gogic model, to achieve the learning results set out by the ICT4E (A1). The inter-vention is defined by five elements:

• Work plan (A8): activities and personnel involved (educational actors and/orexternal team) with the number of person-hours and professional profilerequired according to the design of the intervention.

• Monitoring and evaluation scheme (A9, A13, A14): its objectives are: 1) tomeasure the fidelity of the implementation; 2) to evaluate the outcomes; and 3)to deliver information for decision making during the intervention, especially soas to take remedial action. The fidelity of the implementation defines metrics ofwhen and how the new teaching and learning practices should be manifested inthe classroom. This implies a diagnostic of the necessary pre-existing skills andpractices amongst the actors (which take on the programme in each setting), aswell as following the evolution of the programme’s adoption over time (A13).This will allow for study of the relationship between the variability of the imple-mentation, the achieved learning results and the adoption by teachers (A14).

• Intervention resources (A3): all the resources and supporting materials(e.g. documentation, software, hardware, observation guidelines and softwaremanuals) used by the external team.

• External team (A15): those who carry out the intervention with their respec-tive role (e.g. a distance tutor or field professional) and professional profile(e.g. studies, expected or desirable work experience, prior skills).

• Intervention outcomes (A8, A11): skills and practices of the school’s educa-tional actors to implement the pedagogic model.

2.3. Transference

The transference is the professional development and training of the external teamso that the intervention may be applied autonomously, adapting it according to thecontext of the specific school, following effectiveness standards defined by thedesigners of the ICT4E programmes (transference outcomes). For each of the exter-nal team roles, the previous background must be specified, as well as the new skillsand practices required to perform the role (e.g. how to apply the pedagogic model,carry out intervention activities and apply the assessment instruments from the mon-itoring and evaluation scheme). The transference also includes the evaluation of thefidelity of the intervention carried out by the external team (A15).

2.4. Total cost

The total cost of the ICT4E programme (A7, A10) refers to the cost of implementa-tion, intervention and transference for the duration of the programme.

• Implementation cost: the sum of the marginal costs which the school incursfor participating in the programme: the person-hours of the educational actors,resources (including software), equipment (including hardware), infrastructure,materials and others (e.g. administration costs).


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

• Intervention cost: comprises the fixed and variable costs of the external team.The variable costs depend on the number of person-hours which the externalteam spends on the intervention (including associated materials, expenses andresources), while the fixed costs refer to the general and administrative costs(e.g. management personnel, physical infrastructure) which must be dividedpro-rata according to the number of schools to be serviced and the duration ofthe programme (Margolis et al., 2006).

• Transference cost: the set-up cost of the external team, which is divided pro-rataamongst the participating schools, as are the fixed costs of the external team.

With all of the above, the total cost of the programme is calculated. It can beexpressed in different units such as average cost per school, per classroom and perparticipating student, depending on the comparison to be performed.

2.5. Discussion of the proposed framework

In the proposed framework, the processes and settings defined in sections 2.1 to2.4, cover aspects A1 to A15 identified in the literature as relevant (Table 1). TheICT4E programme model is outlined in Figure 1.

In Figure 1, the block arrows represent the three processes from the ICT4E pro-gramme in its corresponding setting, producing outcomes once each one has beencarried out: learning results in students and the respective skills and practices in theICT and school staff and the external team. Figure 1 also outlines the relationsbetween the main roles of the ICT4E programme’s three processes: firstly, the teach-ers perform the implementation. Secondly, members of the external team and theICT and school staff support each other mutually as they carry out the intervention,

Figure 1. Process involved in the ICT4E programme framework.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

while the external team finally monitors and evaluates the implementation. Next, wewill discuss how our framework meets the objectives set out in section 1.2.

While the design of the TEL-supported pedagogic model can be carried outtogether with the final users through design research methodologies (Bielaczyc,2006; Rodríguez, 2010; Van den Akker, Gravemeijer, McKenney, & Nieveen, 2006),it is very unlikely that the schools will design, implement and scale-up educationaltechnologies, because it is not their role. Thus, it is more probable that an ICT4Eprogramme take the form of an externally developed school reform, which typicallyinvolves a short-term influx of resources, making it especially vulnerable to sustain-ability problems, because assistance to facilitate implementation dissipates over time(Coburn, 2003). To address this problem, the ICT4E programme has at its centre thedefinition of skills and practices needed by the actors in each process, as well as themonitoring and evaluation of its adoption: from the teachers and the school takingcontrol of the programme, to the external team responsible for the intervention.

Our framework establishes an order of precedence for the processes, whichstarts with the selection and training of the external team and which concludes withthe measurement of student learning results. However, all of these professionaldevelopment processes require a prior and explicit definition of the activities in theimplementation and intervention. For this reason, the design of an ICT4Eprogramme must be carried out in reverse order, starting with the expected learningresults and the pedagogic model and finishing with the external team.

With respect to the scalability of the ICT4E programme, the issue of depth iscore to the pedagogic model. The sustainability and shift in reform ownership arethe main aims of the monitoring and evaluation scheme from the interventioncomponent, while the spread is the main concern of the transference component.The ICT4E programme assigns the responsibilities of these last two dimensions tothe external team, which involves designing strategies to achieve these goals rightfrom the design of the ICT4E programme.

Regarding the costing parameters, as well as being consistent with those previ-ously identified in studies of the costs of TEL (Bates, 2005; Laurillard, 2007; Perr-aton et al., 2002; Rumble, 2001; Steinberg, 2004; Twigg, 2002), our frameworkincorporates another two which have not previously been identified: the cost associ-ated with monitoring and evaluation, and the cost of the transference. Our frame-work does not consider the programme’s design costs because these are consideredas sunk costs. We also assume that the evaluation of the design and effectivenesshave already been resolved. This topic is discussed extensively by Rodriguez, Nuss-baum, and Dombrovskaia (2011) through the Evolutionary Development model forICT4E programmes.

3. Describing a specific ICT4E programme: the Eduinnova case

In this section we will illustrate how the proposed framework (Figure 1) allows usto specify the intervention and transference by identifying the skills and practicesrequired by the actors involved for the implementation of the pedagogic model, asestablished by the ICT4E programme. This will be exemplified with Eduinnova, anICT4E programme which uses mobile technology inside the classroom to supportcollaborative learning with mobile devices such as laptops, netbooks or PersonalDigital Assistants (PDAs). These devices are used with a one-to-one computingmodel (Penuel, 2006), but shared by various classrooms throughout the day.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

3.1. Implementation

The purpose of Eduinnova is to support student attainment through computer-sup-ported collaborative learning (CSCL) (Dillenbourg, 1999), constructivist (Zurita &Nussbaum, 2004a) and collaborative learning approaches (Zurita & Nussbaum,2004b). Currently, the programme covers the contents of the curriculum for mathe-matics and Spanish language (third to eighth grade), science and social studies(third and fourth grade) in elementary education; and for mathematics, Spanish lan-guage, physics, chemistry, biology, history and geography in secondary education.

The target schools are elementary and/or secondary urban state-subsidisedschools, from lower-middle socioeconomic strata. In terms of infrastructure, to berun adequately, Eduinnova requires classrooms with at least one power outlet andteacher access to a computer with a broadband connection.

The implementation resources are: the Eduinnova TEL environment (softwarefor laptop or PDA), a laptop (or PDA) equipped with wireless communication (Wi-Fi) for each student and teacher; a wireless router which can support at least 45connections to be used in the classroom; and a cabinet for storing, transporting andcharging the devices’ batteries. In addition, an online library is provided, that: 1)contains CSCL activities previously located by experts within a curriculum frame-work, but customisable by teachers in schools; 2) registers the teachers’ use of thetechnology; and 3) stores the results of the activities undertaken in class. All ofthese resources are used to complement the traditional classroom materials (e.g.books and concrete material).

Next, we will describe the elements of the pedagogic model of Eduinnova fornetbooks/PDAs, to be implemented in a school. Firstly, the way in which the stu-dents learn is outlined in Figure 2.

The teacher accesses an online library in which they can search, create anddownload CSCL activities for their class (Figure 2, step a). The PDAs/netbooksare taken to the classroom in the storage cabinet. In the classroom, using theEduinnova TEL environment, the teacher chooses and sends the CSCL activityto their students (Figure 3, step b) and the class is randomly divided into groupsof three (Figure 2, step c). In each group, the students must solve a problemtogether, with the aid of traditional materials (e.g. books, notes), while theirinteraction is mediated by the TEL environment (Figure 3a). In addition, thesoftware also gives them feedback on the answer and does not allow the stu-dents to progress until they arrive at the solution to the given problem (Cortezet al., 2005; Nussbaum et al., 2009; Rodríguez et al., 2010). At the same time,the TEL environment delivers reports to the teacher in real time about the per-formance of the students, giving them the opportunity to mediate the work doneby each group (Figure 2, step d and Figure 3b). This information is also backedup on the web for later use (Figure 2, step e).

Secondly, in the pedagogic model, the assessment of the learning results consid-ers the formative and summative stages. In the formative stage, the teacher givesfeedback to specific groups or to the whole class during the collaborative work. Inthe summative assessment stage, the students answer online individual tests in real-time, supported by the TEL environment. The teacher can also evaluate studentlearning by administering their own tests.

Thirdly, the roles, skills and practices and educational actors that participate inthe implementation are shown in Table 2.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

To finish the description of the pedagogic model, the classroom is the specificdidactic context where this TEL environment is used, with the students from thatclass and during the school timetable, for the specific subject in which it is applied.

The implementation outcome, which is expected to be achieved following theadoption of the programme, is an effect size above 0.25 in the subjects in which itis applied. This is according to the evidence available on the impact of ICT onlearning (Agodini et al., 2003; Dynarski et al., 2007; Liao & Hao, 2008).

Figure 3. Snapshots of a class which is supported by the collaborative teaching methodEduinnova: (a) general panorama of the classroom, (b) students work together with the helpof the teacher.

Figure 2. Using Eduinnova TEL environment in school.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

Table2.

Roles,skillsandpractices

oftheeducationalactors

intheEduinnova

pedagogicmodel.

Who

Roles

Skills&

practices

Students

Collaborativ

elearning,basedon

4factors(Cortezet

al.,2005):

•Individual

responsibility(S1).

•Mutualsupport(S2).

•Positive

interdependence(S3).

•Individual

responsibility:

each

studentisin

charge

oftheirow

nworkand

effortto

learn.

•Socialface-to-face

interaction(S4).

•Mutualsupport:teaching

theothermem

bers

ofthegroup.

•Content

learning

inspecificsubject(S5).

•Positive

interdependence:

collaboratio

nisonly

successful

whenevery

studenthasinteracted

andaccomplishedtheirindividual

goalsto

solvethe

groupproblem.

•Socialface-to-face

interaction:

thedecision

–makingprocessrelieson

the

ability

toefficiently

exchange

opinions,negotiate

andconstructan

answ

ertogether.

Teachers

•From

‘expert'to

facilitator,mediatorandguide(Condie&

Munro,2007),

supportin

g,monito

ring

andprovidingfeedback

whenstudentswork

collaborativ

ely.

•Technologicalhandlin

gof

theEduinnova

TEL(S6).

•Mediate

students'g

roup

workinside

the

classroom

andmonito

rtheiroutcom

es(S7).

•Integratemobile

CSCLactiv

ities

into

alecturein

accordance

with

the

subject,objectives,tim

ingandresources(digitalor

non-digital)involved.

•Givecontinuous

feedback

basedon

theresults

obtained

(S8).

•Createnew

activ

ities,select

orcustom

iseexistin

gones

from

theonlin

elib

rary.

•IntegrateCSCLmethodology

into

theirow

nteaching

styles

andotheractiv

ities

and

resourcesfrom

theirsubject(S9).

•Sharesmaterials,know

ledgeandexperience

ofcollaborativ

eteaching

practices

with

otherteachers.

•Adapt

digitalresourcesfrom

theprogrammeto

suittheirow

nneeds(S10).

(Contin

ued)


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

Table

2.(Contin

ued).

Who

Roles

Skills&

practices

ICT&

school

staff

•ICTstaff:transport,recharging

devicesandsolvingtechnicalissues

inside

theclassroom.

•Technicalsupportfortheteacher(S11).

•Schoolstaff:planning

andmonito

ring

thecurriculum

andprogress

ofthe

students.

•Managethedevicesin

orderto

guaranteeand

maxim

isetheiraccess

anduse(S12).

•Carry

outmonito

ring

controlandpedagogical

supportto

teacherwhenusingthe

programme(S13).

•Plantheuseof

technology,consistent

with

institu

tionalinterestsandthepedagogical

needswhich

aredetected

(S14).


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

In sections 3.2 to 3.4, we will describe how skills and practices are developedduring the intervention and transference, as well as how its adoption is monitoredand evaluated.

3.2. Intervention

The objectives of the intervention in Eduinnova are: 1) to develop the skills andpractices S1 to S14; 2) to measure its adoption through the systematic applicationof assessment instruments which form part of the monitoring and evaluationscheme; and 3) to take remedial actions in case of excessively large gaps in relationto the expected evolution.

Eduinnova shows that, although the ICT4E programme framework sets an orderof precedence between implementation and intervention, these do not have to bestrictly sequential. The intervention in Eduinnova consists of an implementation withcoaching inside the classroom which looks to develop a hands-on experienceamongst the teachers, with a formative and summative evaluation of their activities,in order to develop skills and practices S6 to S10. Thus, the implementation andintervention overlap to support the teachers in the integration of TEL in their classes.

The work plan of the intervention takes place over two years and is divided intofour main phases and 12 activities, as detailed in Table 3, which also indicates theskills and practices to be developed during each phase.

The key phase of the plan is Phase 2 (‘Training and Coaching’), where theteachers learn to use the TEL environment, both pedagogical and technological, tocarry out the practices of the programme (S6 to S10). It is expected that in thisphase the pedagogic model will be adopted, verified by a participant observation ofthe teachers in their first class using the TEL environment (activity 5). In this obser-vation, any doubts are cleared up and the work done during the face-to-face trainingis reinforced (activity 4). According to the results of this observation, additionalsupport is given to those teachers that have adopted the pedagogic model to a lesserextent. Subsequently, the level of adoption is only measured in the day-to-dayactivity of the teachers with non-participant observation, giving them individualfeedback in writing and detailing their main strengths as well as giving specific sug-gestions (activity 6). When necessary, remedial actions are taken (activity 9). Theother activities from Phase 2 are: giving technical support to schools (activity 7)and monitoring and evaluating the programme (activity 8). Skills and practices S11to S14 are acquired by working with other educational actors within the school.

The monitoring and evaluation scheme has four adoption indicators: teachers’technical skills (S6), pedagogical skills in collaborative learning (S7 and S8), ICTintegration within the curriculum (S9 and S10) and ICT management inside theclassroom (S6 plus efficiency of use of classroom time). The assessment instru-ments are observation protocols based on previously validated rubrics. Studiesindicate that if these indicators do not reach a sufficient final threshold, statisticallysignificant differences are produced in the learning amongst students whose teachershave different levels of adoption (Rodríguez et al., 2010). For further details anddiscussion of this monitoring and evaluation scheme, please refer to Rodriguezet al. (2010).

The most important intervention resources are the assessment instruments of theadoption of the practices, for example, the previously mentioned observation proto-cols (activity 5).


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

Table3.

Workplan

forEduinnova

programme(Rodríguez

etal.,2010).


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

The external team performs, mediates and supports the intervention. The rolesare: project-coordinator professionals who coach teachers inside the classroom(pedagogical assistants) and technicians who support school ICT staff (technicalassistants). The professional profiles of each are the following:

• Project coordinator: educational professional (e.g. teacher or educational psy-chologist) with more than three years of experience and ICT skills at complextools level, as established in UNESCO’s (2008) ICT competency frameworkfor teachers.

• Pedagogical assistants: teacher with more than two years of experience andknowledge of the internal organisation of the schools according to the countrywhere the programme is implemented, and ICT skills at complex tools level.

• Technical assistants: programmer or systems analyst.

In the case of classroom and technical assistants, each assistant works with acertain number of schools depending on the phase of intervention in which eachschool can be found. Thus, the organisational structure of the external team dependson the number of schools participating in the programme.

The intervention outcomes are the skills and practices S1 to S14, defined insection 3.1 (Table 2).

3.3. Transference

The transference is the professional development and training of the external teamin three areas: implementation, intervention and use of technology applied in theprogramme at advanced level. They are also trained in the application of assessmentinstruments from the monitoring and evaluation scheme, so as to guarantee thequality and reproducibility of the programme’s implementation.

3.4. Total cost

The calculation of the total cost is exemplified by calculating the intervention cost.For this, we will take activity 5 (‘Coaching teachers…’) mentioned in section 3.2(Table 3). Activity 5 is divided into two activities. In the first (‘5.1 Observation ofeach teacher’s first class’) the teacher’s work is observed and supported, reinforcingpreviously acquired skills and following an observation guideline after the class. Inthe second activity (‘5.2 Teacher observation…’) the teachers that show most signsof difficulty are further supported. If each observation lasts approximately 1.5 hours,and assuming that a maximum of 15 teachers are used in each school with an aver-age rate of 20% of teachers showing a low level of adoption (of previous experi-ences), 22.5 and 4.5 person-hours are used for activities 5.1 and 5.2, respectively.Considering in Chile a cost of US$9/person-hour (pedagogical supporter profile),material costs (US$7) and transport (US$10), we arrive at a total cost of US$260per school for this activity, given that it is only carried out during the first year (seeTable 3).

Similarly, the variable costs for the remaining activities are calculated. The fixedcosts of the intervention and transference are calculated according to the scale ofthe schools where it is to be implemented.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

4. Current applications of ICT4E programme

The ICT4E programme framework is used as a specification for the ICT curriculumintegration strategies supported by Enlaces (http://www.enlaces.cl), the educationaltechnology agency from the Chilean Ministry of Education (Rodríguez, 2008). Allprogrammes that apply for funding from said agency must adjust themselves to thisframework. To determine if a programme fits the specification, an evaluation standardis applied (Román, 2010). This standard uses the Context Input Processes Products(CIPP) methodology of analysis (Stufflebeam & Shinkfield, 2007) to define the indica-tors for the quality of the design, processes and impact of an ICT4E programme.

When a programme passes the evaluation, it is included in the catalogue of digi-tal educational resources promoted by Enlaces for all schools in Chile (http://www.catalogored.cl). In addition to Eduinnova, there are other ICT4E programmescurrently included in said catalogue, such as those for one-to-one learning (Lagoset al., 2009; López, Parada, & Gómez, 2009), project-based learning (Sánchez &Alarcón, 2003), mathematics (Díaz, 2010), Spanish language (Basáez, Bastías,Villalobos, & Cafena, 2010), science (Miranda & Feris, 2009; Moenne, 2008, 2010)and civic education (Contreras, 2008).

Additionally, the ICT4E programme framework allows for a more accurate esti-mation of the total cost of integrating ICT, because it helps to determine all of thematerials and resources which the programme needs. Such estimations enable ananalysis of the economic feasibility of the programme, determining whether or not itcan be implemented given the financial restrictions on school and Ministry of Educa-tion spending (Margolis et al., 2006). Another possible application is to compareprogrammes using a cost-effectiveness analysis (Belli, Anderson, Barnum, Dixon, &Tan, 2001; Creemers & van der Werf, 2000; Feinstein & Picciotto, 2000; Harris,2009; Margolis et al., 2006). The ICT4E programme model offers a cost-comparisonframework, standardising the process components and resource prices of the pro-grammes under comparison. For example, the definition of the profiles for the exter-nal team facilitates the comparison between programmes as professionals withsimilar training and experience should have similar costs within the same country.

The ICT4E programme supported the design of the Eduinnova programme,which, in the last decade, has been tested in different grade levels and with differentsubjects: preschool education (Infante et al., 2009), elementary schools (Zurita &Nussbaum, 2004b), secondary schools (Rodríguez et al., 2010) and university level(Bustos & Nussbaum, 2009; Valdivia & Nussbaum, 2007, 2009). The transferencecomponent allowed Eduinnova to also be implemented in schools in Argentina,Brazil, Guatemala, Great Britain (Nussbaum et al., 2009) and the United States(Roschelle et al., 2010). The effect sizes found in quasi-experimental studies fluctu-ate between 0.32 and 0.41 (Rodríguez et al., 2010, 2011), consistent with theexpected learning results for an educational technology (section 3.1).

Finally, another nonetheless important application was the development of amodel for the design, implementation and evaluation of ICT4E programmes. Saidmodel – called Evolutionary Development – builds each component of the ICT4Eprogramme in stages and validates it in real educational settings through formativeand summative evaluations based on design research methodologies and usingexperimental or quasi-experimental studies (Rodríguez et al., 2011). Thus, theefficacy of the implementation, the effectiveness of intervention and the efficiencyof the programme as a whole are tested empirically.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

http://www.enlaces.cl

http://www.catalogored.cl

http://www.catalogored.cl

5. Conclusions and future work

In this article, we proposed the ICT4E programme framework to respond to thequestion about what an educational programme supported by ICT should look like(section 2). We centred the framework on producing a sustainable change in timethrough the effective adoption of new teaching and learning practices supportedby TEL environments. The key components of an ICT4E programme are: theimplementation of the TEL-supported pedagogic model, intervention, transferenceand total costs. The transference component comprises the formation strategies ofan external team to guarantee the replicability and quality of the intervention.

The proposed framework (objective 1 of the article) makes explicit the relation-ships between each of the activities and the expected outcomes (at an intermediateand final level) in each process of the ICT4E programme (objective 2). Secondly, itallows educators to identify which elements (e.g. skills and practices, actorsinvolved, activities, resources) can be absent (objective 3). Finally, it enables thecalculation of the real total cost of the programme (objective 4). These featureswere illustrated by the Eduinnova programme in section 3.

Currently, in Chile, the ICT4E programme model is applied to specify ICT pro-jects in education subject to public funding and evaluate them according to a stan-dard built using the framework as a base (Román, 2010). Other applications are theanalysis of economic feasibility and cost-effectiveness (Margolis et al., 2006). Thislast case presents practical challenges as the effectiveness must be measured undercriteria from rigorous evidence (Slavin, 2008). Furthermore, only little is knownabout how the effectiveness and costs of an ICT4E programme vary with time,especially with respect to long-term substitution of obsolete technology.

Thus, a future challenge for policy making is to determine how the cost-effec-tiveness varies in the medium and long term, and how such changes influence thecurrent comparison between programmes. To resolve this problem, some authorspropose the calculation of a present value both of the effect size as well as of thecosts, which would imply making suppositions with respect to the discount rateto be applied and the time horizon for the analysis (Harris, 2009). One empiricalsolution would be to carry out longitudinal studies of the population affected byICT4E programmes to see if the effects are sustainable, and determine the rate atwhich they decline in the long term. Periodical cross-sectional studies would alsoverify whether or not an ICT4E programme systematically produces the expectedoutcomes. If it does not, the ICT4E programme model can conduct an audit todiscover which aspects are loss-making and to what extent.

Another form of comparing educational programmes is through cost–benefitstudies, which are harder to carry out because the benefits must be monetarilyquantified (Belli et al., 2001; Margolis et al., 2006). However, Laurillard (2007)proposed a prospective method which allows the identification of specific benefitsof a TEL environment. Our ICT4E programme supports the implementation of saidmethod, as it facilitates the calculation of the costs and also specifies the intermedi-ary and final outcomes which should be achieved. Thus, after valuing the identifiedcosts and benefits, it would be possible to calculate the net present value and returnof investment of implementing an ICT4E programme in a school or district. Withthese indexes policy makers have more quantitative information with which to com-pare other educational alternatives (with or without the technology) being availableto them at the time of evaluating.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

The specification of processes and calculation of the costs that the ICT4Eprogramme enables also allows the application of process improvement and cost-reduction models – such as Capability Maturity Model Integration (Forrester, Buteau,& Shrum, 2011) – in order to improve the cost-effectiveness of the programme.

Another aspect to be determined is whether or not there exist significant differ-ences in the impact of ICT4E programmes which adjust more to the framework, incomparison to those that do it to a lesser degree, a topic which is the subject ofresearch currently being conducted.

Finally, there are at least two issues for further exploration. Firstly, the proposedframework could be extended to include other aspects of ICT in education bysubstituting the pedagogic model with one designed to support such processes (e.g.,school administration and learning management models). Secondly, our ICT4E pro-gramme framework has the potential to specify less teacher-centred models, likelifelong learning or distance education, by redefining the concept of ‘settings’within the framework. For example, the ‘classroom’ setting could be extended toother places outside the school such as the home or office.

AcknowledgementsThis research was funded by the Chilean Ministry of Education through the Enlaces Centerfor Educational Technology, and CONICYT under grants FONDECYT 11110424, CIE-05CIAE and CIE-01 CEPPE. Special thanks go to Claudia Morchio and Goryet Pandorfa fromEnlaces, and Eugenio Severin from the Inter-American Development Bank. This paper isdedicated to the memory of Dr Gerardo Moenne, from the Instituto de InformáticaEducativa, Universidad de la Frontera, Temuco, Chile.

Notes on contributorsPatricio Rodriguez is research associate at the Center of Advanced Research in Education ofthe University of Chile. He has more than 15 years’ experience of educational technology ineducation, with projects implemented in Argentina, Brazil, Chile, United Kingdom andUnited States.

Miguel Nussbaum is full professor for Computer Science at the School of Engineering ofthe Universidad Católica de Chile. He carries out research in instructional design based ontechnology.

Lioubov Dombrovskaia is assistant professor at the Department of Informatics of theUniversidad Técnica Federico Santa Maria in Santiago, Chile. She teaches and carries outresearch in human–computer interaction, software design, bioinformatics, engineering educationand technological innovation.

ReferencesAgodini, R., Dynarski, M., Honey, M., & Levin, D. (2003). The effectiveness of educational

technology: Issues and recommendations for the national study. Mathematica PolicyResearch. Retrieved from http://www.mathematica-mpr.com/PDFs/edtechrec.pdf

Bakia, M., Means, B., Gallagher, L., Chen, E., & Jones, K. (2009). Evaluation of theEnhancing Education Through Technology program: Final report. US Department ofEducation, Office of Planning, Evaluation and Policy Development, Policy and ProgramStudies Service. Washington, DC. Retrieved from http://www2.ed.gov/rschstat/eval/tech/netts/finalreport.pdf

Balanskat, A., Blamire, R., & Kefala, S. (2006). The ICT impact report: A review of studiesof ICT impact on schools in Europe. Retrieved from European Commission website:http://ec.europa.eu/education/pdf/doc254_en.pdf


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

http://www.mathematica-mpr.com/PDFs/edtechrec.pdf

http://www2.ed.gov/rschstat/eval/tech/netts/finalreport.pdf

http://www2.ed.gov/rschstat/eval/tech/netts/finalreport.pdf

http://ec.europa.eu/education/pdf/doc254_en.pdf

Barrera-Osorio, F., & Linden, L. (2009). The use and misuse of computers in education: Evi-dence from a randomized controlled trial of a language arts program. Retrieved fromPoverty Action Lab website: http://www.povertyactionlab.org/evaluation/use-and-misuse-computers-education-evidence-randomized-controlled-trial-language-arts-pro

Basáez, C., Bastías, V., Villalobos, R., & Cafena, D. (2010). Piloto @escribo, usando lasTIC para potenciar la escritura: Informe final [@escribo, using ICT to improve writing.Final report]. Retrieved from Enlaces website: http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3709

Baskin, C., & Williams, M. (2006). ICT integration in schools: Where are we now and whatcomes next? Australasian Journal of Educational Technology, 22, 455–473.

Batchelor, S., & Norrish, P. (2005). Framework for the assessment of ICT pilot projects.Retrieved from infoDev website: http://www.infodev.org/en/Publication.4.html

Bates, A. (2005). Technology, e-learning and distance education (2nd ed.). London: Routledge.Belli, P., Anderson, J.R., Barnum, H.N., Dixon, J.A., & Tan, J.-P. (2001). Economic

analysis of investment operations. Analytical tools and practical applications. Wash-ington, DC: The International Bank for Reconstruction and Development/The WorldBank.

Bielaczyc, K. (2006). Designing social infrastructure: Critical issues in creating learningenvironments with technology. Journal of the Learning Sciences, 15, 301–329.

Burns, T.C., & Ungerleider, C.S. (2003). Information and communication technologies inelementary and secondary education. International Journal Educational Policy, Researchand Practice, 3(4), 27–54.

Bustos, H., & Nussbaum, M. (2009). An experimental study of the inclusion of technologyin higher education. Computer Applications in Engineering Education, 17, 100–107.

Callister, T.A. (1992). The computer as doorstep: Technology as disempowerment. Phi DeltaKappan, 74, 324–329.

Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveness of reading andmathematics software products: Findings from two student cohorts. Retrieved from Insti-tute of Education Sciences website: http://ies.ed.gov/ncee/pubs/20094041/

Cartwright, V., & Hammond, M. (2007). ‘Fitting it in’: A study exploring ICT use in a UKprimary school. Australasian Journal of Educational Technology, 23, 390–407.

Castro, C. (2004). Are new technologies better technologies? For whom? In D.W. Chap-man & L.O. Mählck (Eds.), Adapting technology for school improvement: A globalperspective (pp. 39–54). Paris: International Institute for Educational Planning,UNESCO.

Cloke, C., & Sharif, S. (2001). Why use information and communications technology? Sometheoretical and practical issues. Technology, Pedagogy and Education, 10, 7–18.

Coburn, C.E. (2003). Rethinking scale: Moving beyond numbers to deep and lasting change.Educational Researcher, 32(6), 3–12.

Condie, R., & Munro, B. (2007). The impact of ICT in schools: A landscape review. Coven-try: Becta.

Conlon, T., & Simpson, M. (2003). Silicon Valley versus Silicon Glen: The impact of com-puters upon teaching and learning: A comparative study. British Journal of EducationalTechnology, 34, 137–150.

Contreras, D. (2008). [email protected], alfabetización digital para ciudadanos en red: Informe final[[email protected], digital literacy for networked citizens. Final report]. Retrieved from Enlaceswebsite: http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=2296

Cortez, C., Nussbaum, M., López, X., Rodríguez, P., Santelices, R., Rosas, R., et al. (2005).Teachers’ support with ad-hoc collaborative networks. Journal of Computer AssistedLearning, 21, 171–180.

Cox, M. (2008). Research IT in education. In J. Voogt & G. Knezek (Eds.), Internationalhandbook of information technology in primary and secondary education (pp. 965–981).New York, NY: Springer.

Cox, M., & Abbot, C. (2004). ICT and attainment: A review of the research literature. ICTin Schools Research and Evaluation Series – No. 17, Department for Education andSkills and Becta.

Cox, M., & Marshall, G. (2007). Effects of ICT: Do we know what we should know? Edu-cation and Information Technologies, 12(2), 59–70.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

http://www.povertyactionlab.org/evaluation/use-and-misuse-computers-education-evidence-randomized-controlled-trial-language-arts-pro

http://www.povertyactionlab.org/evaluation/use-and-misuse-computers-education-evidence-randomized-controlled-trial-language-arts-pro

http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3709


http://www.infodev.org/en/Publication.4.html

http://ies.ed.gov/ncee/pubs/20094041/


Cox, M., & Webb, M. (2004). ICT and pedagogy: A review of the research literature. ICTin Schools Research and Evaluation Series – No. 18, Department for Education andSkills and Becta.

Creemers, B., & van der Werf, G. (2000). Economic viewpoints in educational effectiveness:Cost-effectiveness analysis of an educational improvement project. School Effectivenessand School Improvement, 11, 361–384.

Cuban, L. (1986). Teachers and machines: The classroom use of technology since 1920.New York, NY: Teachers College Press, Columbia University.

Culp, K.M., Honey, M., & Mandinach, E. (2003). A retrospective on twenty years of educa-tion technology policy. US Department of Education, Office of Educational Technology.Retrieved from http://www.ed.gov/rschstat/eval/tech/20years.pdf

Dede, C. (2006). Scaling up: Evolving innovations beyond ideal settings to challenging con-texts of practice. In R.K. Sawyer (Ed.) The Cambridge Handbook of Learning Sciences(pp. 551–566). Cambridge: Cambridge University Press.

Díaz, W. (2010). Proyecto Enlaces Matemática: Informe final. Retrieved from http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=4011

Dillenbourg, P. (1999). Collaborative learning: Cognitive and computational approaches.Advances in learning and instruction series. Elsevier Science, New York.

Dillenbourg, P. (2008). Integrating technologies into educational ecosystems. DistanceEducation, 29, 127–140.

Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., et al.(2007). Effectiveness of reading and mathematics software products: Findings from thefirst student cohort. US Department of Education, Institute of Education Sciences.Retrieved from http://ies.ed.gov/ncee/pubs/20074005/

Earle, R.S. (2002). The integration of instructional technology into public education: Prom-ises and challenges. Educational Technology, 42, 5–13.

Feinstein, O.N., & Picciotto, R. (2000). Evaluation and poverty reduction. Proceedings from aWorld Bank conference. World Bank Series on Evaluation and Development (Vol. 3). Wash-ington, DC: The International Bank for Reconstruction and Development/The World Bank.

Forrester, E., Buteau, B., & Shrum, S. (2011). CMMI for services: Guidelines for superiorservice (2nd ed.). Boston, MA: Addison-Wesley.

Harris, D.N. (2009). Toward policy-relevant benchmarks for interpreting effect sizes: Com-bining effects with costs. Educational Evaluation and Policy Analysis, 31, 3–29.

Hayes, D.N.A. (2007). ICT and learning: Lessons from Australian classrooms. Computers &Education, 49, 385–395.

Hennessy, S., Ruthven, K., & Brindley, S. (2005). Teacher perspectives on integrating ICTinto subject teaching: Commitment, constraints, caution, and change. Journal of Curricu-lum Studies, 37, 155–192.

Infante, C., Weitz, J., Reyes, T., Nussbaum, M., Gómez, F., & Radovic, D. (2010). Co-located collaborative learning video game with single display groupware. InteractiveLearning Environments, 18, 2, 177–195.

Kikis, K., Scheuermann, F., & Villalba, E. (2009). A framework for understanding and eval-uating the impact of information and communication technologies in education. In F.Scheuermann & F. Pedró (Eds.), Assessing the effects of ICT in education: Indicators,criteria and benchmarks for international comparisons (pp. 69–82). Luxembourg: Publi-cations Office of the European Union.

Kirkman, C. (2000). A model for the effective management of information and communica-tions technology development in schools derived from six contrasting case studies. Tech-nology, Pedagogy and Education, 9, 37–52.

Kozma, R.B. (1991). Learning with media. Review of Educational Research, 61, 179–211.Kozma, R.B. (2003). Technology and classroom practices: An international study. Journal of

Research on Technology in Education, 36, 1–14.Kulik, J. (2003). Effects of using instructional technology in elementary and secondary

schools: What controlled evaluation studies say. Technical report, SRI International.Lagos, M.E., Vil, G., Miranda, H., Contreras, D., Rodríguez, J., & Sandoval, K. (2009).

Aprendiendo Matemática y Lenguaje con Tecnología Portátil 1:1: Informe final [Learn-ing mathematics and language using 1:1 computing. Final report]. Retrieved from Enlac-es website: http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3626


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

http://www.ed.gov/rschstat/eval/tech/20years.pdf



http://ies.ed.gov/ncee/pubs/20074005/


Lai, K.-W. (2008). ICT supporting the learning process: The premise, reality, and promise.In J. Voogt & G. Knezek (Eds.), International handbook of information technology inprimary and secondary education (pp. 215–230). New York, NY: Springer.

Laurillard, D. (2007). Modelling benefits-oriented costs for technology enhanced learning.Higher Education, 54, 21–39.

Liao, Y.C., & Hao, Y. (2008). Large-scale studies and quantitative methods. In J. Voogt &G. Knezek (Eds.), International handbook of information technology in primary and sec-ondary education (pp. 1019–1035). New York, NY: Springer.

Light, D. (2008). Evaluating educational technology interventions: How do we know it isworking? Center for Children and Technology, Education Development Center. Retrievedfrom http://cct.edc.org/report.asp?id=254

Lim, C.P. (2002). A theoretical framework for the study of ICT in schools: A proposal. Brit-ish Journal of Educational Technology, 33, 411–421.

Lim, C.P., & Hang, D. (2003). An activity theory approach to research of ICT integration inSingapore schools. Computers & Education, 41, 49–63.

López, M., Parada, D., & Gómez, F. (2009). MIC (1:1) Modelo de Integración Curricularcon estrategia 1 a 1. Resultados de la Implementación: Informe final [Curricular integra-tion model with 1:1 computing. Final results]. Retrieved from Enlaces website: http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3677

Luckin, R. (2008). The learner centric ecology of resources: A framework for using technol-ogy to scaffold learning. Computers & Education, 50, 449–462.

Margolis, J., Nussbaum, M., Rodríguez, P., & Rosas, R. (2006). Methodology for evaluatinga novel education technology: A case study of handheld video games in Chile. Comput-ers & Education, 46, 174–191.

Marshall, G., & Cox, M. (2008). Research methods: Their design, applicability and reliabil-ity. In J. Voogt & G. Knezek (Eds.), International handbook of information technologyin primary and secondary education (pp. 983–1002). New York, NY: Springer.

McDonald, S.-K., Keesler, V.A., Kauffman, N.J., & Schneider, B. (2006). Scaling-up exem-plary interventions. Educational Researcher, 35(3), 15–24.

McFarlane, A. (2001). Perspectives on the relationships between ICT and assessment. Jour-nal of Computer Assisted Learning, 17, 227–234.

Mioduser, D., Nachmias, R., Tubin, D., & Forkosh-Baruch, A. (2003). Analysis schema forthe study of domains and levels of pedagogical innovation in schools using ICT. Educa-tion and Information Technologies, 8, 23–36.

Miranda, S., & Feris, C. (2009). Modelo de Informática Educativa Aprendizaje Interactivode las Ciencias. Enseñanza Media: Informe Final [Interactive learning of science forsecondary students. Final report]. Retrieved from Enlaces website: http://intranet.reden-laces.cl/index.php?id=11414&no_cache=1&uid=3740

Moenne, G. (2008). Dibujos con Vida. Informe final [Drawings alive. Final report].Retrieved from Enlaces website: http://intranet.redenlaces.cl/index.php?id=11414&no_ca-che=1&uid=2186

Moenne, G. (2010). Atenea: Desarrollo del pensamiento científico a través de las TIC: Informefinal [Development of scientific thinking using ICT. Final report]. Retrieved from Enlaceswebsite: http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3839

Mooij, T., & Smeets, E. (2001). Modelling and supporting ICT implementation in secondaryschools. Computers & Education, 36, 265–281.

Moonen, J. (2008). Policy from a global perspective. In J. Voogt & G. Knezek (Eds.),International handbook of information technology in primary and secondary education(pp. 1171–1178). New York, NY: Springer.

Moyle, K. (2008). Total cost of ownership and total value of ownership. In J. Voogt & G.Knezek (Eds.), International handbook of information technology in primary and sec-ondary education (pp. 615–631). New York, NY: Springer.

Nachmias, R., Mioduser, D., Cohen, A., Tubin, D., & Forkosh-Baruch, A. (2004). Factorsinvolved in the implementation of pedagogical innovations using technology. Educationand Information Technologies, 9, 291–308.

Nussbaum, M., Alvarez, C., McFarlane, A., Gomez, F., Claro, S., & Radovic, D. (2009).Technology as small group face-to-face collaborative scaffolding. Computers & Educa-tion, 52, 147–153.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

http://cct.edc.org/report.asp?id=254








Pelgrum, W. (2001). Obstacles to the integration of ICT in education: Results from a world-wide educational assessment. Computers & Education, 37, 163–178.

Penuel, W.R. (2005). Recommendations for evaluation research within educational innova-tions. Center for Technology in learning, SRI International. Retrieved from http://ctl.sri.com/publications/downloads/Rec_Eval_Research_within_Edu_Innovations.pdf

Penuel, W.R. (2006). Implementation and effects of one-to-one computing initiatives: Aresearch synthesis. Journal of Research on Technology in Education, 38, 329–348.

Perraton, H., Creed, C., & Robinson, B. (2002). Teacher education guidelines: Using openand distance learning. Technology, curriculum, cost, evaluation. UNESCO, Higher Edu-cation Division, Teacher Education Section.

Pilkington, R.M. (2008). Measuring the impact of information technology on students’ learn-ing. In J. Voogt & G. Knezek (Eds.), International handbook of information technologyin primary and secondary education (pp. 1003–1018). New York, NY: Springer.

Plomp, T., Pelgrum, W., & Law, N. (2007). SITES2006 – International comparative surveyof pedagogical practices and ICT in education. Education and Information Technologies,12, 83–92.

Reeves, T. (2008). Evaluation of the design and development of IT tools in education. In J.Voogt & G. Knezek (Eds.), International handbook of information technology in primaryand secondary education (pp. 1037–1051). New York, NY: Springer.

Reiser, R.A. (2001). A history of instructional design and technology: Part I: A history ofinstructional media. Educational Technology Research and Development, 49, 53–64.

Reynolds, D., Treharne, D., & Tripp, H. (2003). ICT – The hopes and the reality. BritishJournal of Educational Technology, 34, 151–167.

Robertson, J. (2002). The ambiguous embrace. Twenty years of IT (ICT) in UK primaryschools. British Journal of Educational Technology, 33, 403–409.

Roblyer, M.D. (2005). Educational technology research that makes a difference. Series intro-duction. Contemporary Issues in Technology and Teacher Education, 5, 192–201.

Rodríguez, P. (2008). Estándar de Modelo de Informática Educativa [Standard for an ICT foreducation programme]. Enlaces, Center for Educational Technology. Ministry of Education.Retrieved from http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=2549

Rodríguez, P. (2010). A methodology for replication, scaling-up and transference of ICT sup-ported pedagogical models (PhD thesis). Department of Computer Science, College ofEngineering, Pontificia Universidad Católica de Chile.

Rodríguez, P., Nussbaum, M., & Dombrovskaia, L. (2012). Evolutionary development: amodel for the design, implementation and evaluation of ICTs for education programs.Journal Computer Assisted Learning, 28, 81–98.

Román, M. (2010). Desarrollo de un Estándar de Evaluación para los Modelos de Informá-tica Educativa (MIE): Informe final. Retrieved from http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=4087

Roschelle, J., Rafanan, K., Bhanot, R., Estrella, G., Penuel, W., Nussbaum, M., et al. (2010).Scaffolding group explanation and feedback with handheld technology: Impact on students’mathematics learning. Educational Technology Research and Development, 58, 399–419.

Rumble, G. (2001). Analysing costs/benefits for distance education programmes. The Com-monwealth of Learning. Retrieved from http://www.col.org/PublicationDocuments/pub_KS2001-02_costs.pdf

Sánchez, J., & Alarcón, P. (2003). Multimedia Project en Chile: Diseño y resultados [Multi-media project in Chile: design and results]. Presented at the Taller Internacional de Soft-ware Educativo, Santiago, Chile. Retrieved from Multimedia Project website: http://mmpchile.c5.cl/pag/publi/multimedia_project_diseno_resultados.pdf

Scrimshaw, P. (2002). Total cost of ownership: A review of the literature. London: Depart-ment for Education and Skills.

Slavin, R.E. (2008). Perspectives on evidence-based research in education – What works?Issues in synthesizing educational program evaluations. Educational Researcher, 37,5–14.

Snider, R.C. (1992). The machine in the classroom. Phi Delta Kappan, 74, 316–323.Steinberg, D.C. (2004). The business model of e-learning in UK higher education. Industry

and Higher Education, 18, 187–198.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

http://ctl.sri.com/publications/downloads/Rec_Eval_Research_within_Edu_Innovations.pdf

http://ctl.sri.com/publications/downloads/Rec_Eval_Research_within_Edu_Innovations.pdf




http://www.col.org/PublicationDocuments/pub_KS2001-02_costs.pdf

http://www.col.org/PublicationDocuments/pub_KS2001-02_costs.pdf

http://mmpchile.c5.cl/pag/publi/multimedia_project_diseno_resultados.pdf

http://mmpchile.c5.cl/pag/publi/multimedia_project_diseno_resultados.pdf

Stufflebeam, D.L., & Shinkfield, A.J. (2007). Evaluation theory, models, and applications.San Francisco, CA: John Wiley and Sons.

Sunkel, G. (2006). Las Tecnologías de La Información y la Comunicación (Tic) en la educa-ción en América Latina. Una exploración de indicadores [ICT in Latin America: explor-ing indicators]. CEPAL – Politicas sociales (Division de Desarrollo Social, CEPAL,Naciones Unidas). United Nations.

Sutherland, R., Armstrong, V., Barnes, S., Brawn, R., Breeze, N., Gall, M., et al. (2004).Transforming teaching and learning: Embedding ICT into everyday classroom practices.Journal of Computer Assisted Learning, 20, 413–424.

Tearle, P. (2003). ICT implementation: What makes the difference? British Journal of Edu-cational Technology, 34, 567–583.

Tearle, P. (2004). A theoretical and instrumental framework for implementing change in ICTin education. Cambridge Journal of Education, 34, 331–351.

Tondeur, J., van Braak, & Valcke, M. (2007). Curricula and the use of ICT in education:Two worlds apart? British Journal of Educational Technology, 38, 962–976.

Tondeur, J., Valcke, M., & van Braak, J. (2008). A multidimensional approach to determi-nants of computer use in primary education: Teacher and school characteristics. Journalof Computer Assisted Learning, 24, 494–506.

Toyama, K. (2010). Can technology end poverty? Boston Review (November/December).Retrieved from http://bostonreview.net/BR35.6/ndf_technology.php

Toyama, K. (2011). There are no technology shortcuts to good education. EducationalTechnology Debate, infoDev-UNESCO. Retrieved from http://edutechdebate.org/ict-in-schools/there-are-no-technology-shortcuts-to-good-education/

Trucano, M. (2005). Knowledge maps: ICTs in education. ICT and Education Series.Washington, DC: infoDev/World Bank. Retrieved from http://www.infodev.org/en/Publi-cation.8.html

Tubin, D., Mioduser, D., Nachmias, R., & Forkosh-Baruch, A. (2003). Domains and levelsof pedagogical innovation in schools using ICT: Ten innovative schools in Israel. Educa-tion and Information Technologies, 8, 127–145.

Twigg, C. (2002). Improving quality and reducing costs: Designs for effective learning usingICT. Observatory on Borderless Higher Education.

UNESCO. (2008). ICT competency standards for teachers – Competency standards modules.Paris: UNESCO. Retrieved from http://cst.unesco-ci.org/sites/projects/cst/The%20Stan-dards/ICT-CST-Competency%20Standards%20Modules.pdf

Valdivia, R., & Nussbaum, M. (2007). Face-to-face collaborative learning in computer sci-ence classes. International Journal of Engineering Education, 23, 434–440.

Valdivia, R., & Nussbaum, M. (2009). Using multiple choice questions as a pedagogic modelfor face-to-face CSCL. Computer Applications in Engineering Education, 17, 89–99.

Van den Akker, J. (2003). Curriculum perspectives: An introduction. In J. van den Akker,U. Hameyer, & W. Kuiper (Eds.), Curriculum landscapes and trends (pp. 1–10). Dordr-echt: Kluwer.

Van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N. (2006). Educationaldesign research. London: Routledge.

Wagner, D.A., Day, B., James, T., Kozma, R.B., Miller, J., & Unwin, T. (2005). Monitoringand evaluation of ICT in education projects: A handbook for developing countries. ICTand Education Series. Washington, DC: infoDev/World Bank. Retrieved from http://www.infodev.org/en/Publication.9.html

Warschauer, M., Knobel, M., & Stone, L. (2004). Technology and equity in schooling:Deconstructing the digital divide. Educational Policy, 18, 562–588.

Watson, D.M. (2001). Pedagogy before technology: Re-thinking the relationship betweenICT and teaching. Education and Information Technologies, 6, 251–266.

Waxman, H., Lin, M., & Michko, G.M. (2003). Meta-analysis: Effects of educational tech-nology on student outcomes. Learning Point Associates. Retrieved from http://www.ncrel.org/tech/effects2/

Zurita, G., & Nussbaum, M. (2004a). A constructivist mobile learning environment supportedby a wireless handheld network. Journal of Computer Assisted Learning, 20, 235–243.

Zurita, G., & Nussbaum, M. (2004b). Computer supported collaborative learning using wire-lessly interconnected handheld computers. Computers & Education, 42, 289–314.


Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 1

3:56

03

Oct

ober

201

3

http://bostonreview.net/BR35.6/ndf_technology.php

http://edutechdebate.org/ict-in-schools/there-are-no-technology-shortcuts-to-good-education/

http://edutechdebate.org/ict-in-schools/there-are-no-technology-shortcuts-to-good-education/



http://cst.unesco-ci.org/sites/projects/cst/The%20Standards/ICT-CST-Competency%20Standards%20Modules.pdf

http://cst.unesco-ci.org/sites/projects/cst/The%20Standards/ICT-CST-Competency%20Standards%20Modules.pdf



http://www.ncrel.org/tech/effects2/

http://www.ncrel.org/tech/effects2/