ICT for education: a conceptual framework for the sustainable adoption of technology-enhanced learning environments in schools

  • Published on
    09-Dec-2016

  • View
    213

  • Download
    1

Transcript

  • This article was downloaded by: [University of Birmingham]On: 03 October 2013, At: 13:56Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

    Technology, Pedagogy and EducationPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/rtpe20

    ICT for education: a conceptualframework for the sustainable adoptionof technology-enhanced learningenvironments in schoolsPatricio Rodrguez a , Miguel Nussbaum b & Lioubov Dombrovskaiac

    a Center for Advanced Research in Education, Universidad deChile, Santiago, Chileb Department of Computer Science, Pontificia UniversidadCatlica de Chile, Santiago, Chilec Informatics Department, Universidad Tcnica Federico SantaMara, Santiago, ChilePublished online: 28 Sep 2012.

    To cite this article: Patricio Rodrguez , Miguel Nussbaum & Lioubov Dombrovskaia (2012) ICTfor education: a conceptual framework for the sustainable adoption of technology-enhancedlearning environments in schools, Technology, Pedagogy and Education, 21:3, 291-315, DOI:10.1080/1475939X.2012.720415

    To link to this article: http://dx.doi.org/10.1080/1475939X.2012.720415

    PLEASE SCROLL DOWN FOR ARTICLE

    Taylor & Francis makes every effort to ensure the accuracy of all the information (theContent) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.

    This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,

  • systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • ICT for education: a conceptual framework for the sustainableadoption of technology-enhanced learning environments in schools

    Patricio Rodrgueza*, Miguel Nussbaumb and Lioubov Dombrovskaiac

    aCenter for Advanced Research in Education, Universidad de Chile, Santiago, Chile;bDepartment of Computer Science, Ponticia Universidad Catlica de Chile, Santiago,Chile; cInformatics Department, Universidad Tcnica Federico Santa Mara, Santiago, Chile

    (Received 26 October 2011; nal version received 23 March 2012)

    Currently, the use of information and communication technologies (ICT) ineducation does not conclusively demonstrate signicant effects on learning.However, not all ICT usage models are designed to affect student outcomes.Therefore, to accurately study the impact of ICT, the concept of an educationalprogramme supported by ICT must rst be dened. The authors propose theICT for Education (ICT4E) programme, an evidence-based framework to deter-mine a models ability to produce improvements before having to evaluate itsresults. The framework has four components: implementation, intervention,transference and total cost. Based on an explicit denition of the outcomes pur-sued by a given programme, this framework supports the design of its activitiesto promote sustainable changes in pedagogical practices in schools and calculatethe programmes total cost. The authors illustrate this with a programme imple-mented in six countries. In Chile, the ICT4E programme forms part of the pol-icy for educational technology, evaluating the effectiveness and scalability ofsuch initiatives.

    Keywords: technology-enhanced learning; pedagogic model; ICT effects; scala-bility; sustainability; ICT integration

    1. Introduction

    Attempts to improve education with modern machines such as the phonograph,lantern slides and television date from the early twentieth century (Castro, 2004;Cuban, 1986; Earle, 2002; Reiser, 2001; Snider, 1992). For the most part suchinventions come and go, yet what happens in the classroom looks pretty much thesame (Callister, 1992). Information and communication technologies (ICT) werealso introduced in schools to transform teaching and learning processes and toimprove strategies for better educational attainment (Culp, Honey, & Mandinach,2003; Kozma, 2003; Sunkel, 2006).

    Nonetheless, such desired effects are far from becoming a reality (Moonen,2008; Reynolds, Treharne, & Tripp, 2003; Robertson, 2002; Tondeur, van Braak, &Valcke, 2007; Trucano, 2005). A wide range of ICT applications and uses wereanalysed in several impact assessments: programme evaluations, specic educationalsoftware in particular subjects at large scale, research and case reviews, and

    *Corresponding author. Email: prodriguez@ciae.uchile.cl

    Technology, Pedagogy and EducationVol. 21, No. 3, October 2012, 291315

    ISSN 1475-939X print/ISSN 1747-5139 online 2012 Association for Information Technology in Teacher Educationhttp://dx.doi.org/10.1080/1475939X.2012.720415http://www.tandfonline.com

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • meta-analysis research (see further details in section 2). Some of these showed apositive impact, while others a neutral or negative impact, leading many authors toargue that there is still no conclusive answer to the question about the impact andeffectiveness of ICT in education (Burns & Ungerleider, 2003; Cartwright & Ham-mond, 2007; Cox & Marshall, 2007; Kikis, Scheuermann, & Villalba, 2009;Reeves, 2008; Trucano, 2005). Other authors even argue that the introduction ofICT in schools served to amplify existing forms of inequality (Toyama, 2010, 2011;Warschauer, Knobel, & Stone, 2004).

    However, ICT was not devised with education in mind, it did not appear naturallyin educational systems, nor was it demanded by the teaching community (Sunkel,2006). It responds to a model with its concepts, discourse and practices whichoriginated outside of educational systems (Kikis et al., 2009; Sunkel, 2006) and wasthen imposed upon them (Hennessy, Ruthven, & Brindley, 2005), making theintegration a highly difcult process. This later worsened when greater emphasis wasplaced on technology than on pedagogy (Trucano, 2005; Watson, 2001), ignoring theextensive existing research on teaching and learning (Sutherland et al., 2004).

    1.1. Problem denition

    When ICT were rst being introduced in schools, they were considered so newthat some policy makers and practitioners ignored theoretical foundations central toall learning with or without ICT (Sutherland et al., 2004). In addition to learningtheories, other design aspects of curriculum integration were not considered, suchas the type of ICT resources and how they are specically used in a particular sub-ject (Cox & Marshall, 2007; McFarlane, 2001) and the design and curriculumimplementation of ICT use in the classroom (Cox & Abbot, 2004; Cox & Marshall,2007; McFarlane, 2001).

    By studying ICT integration in schools, it was determined that the way in whichthis takes place depends on the educational context. As part of this context, thefollowing factors were also identied: pedagogical approaches and beliefs (Coburn,2003; Cox & Webb, 2004; Kirkman, 2000; Trucano, 2005), teacher condence, atti-tudes and skills relating to ICT; school ICT infrastructure, supervision and technicalsupport, involvement and leadership of school principals and time spent by teacherson meetings, training, exercises and lesson planning (Baskin & Williams, 2006;Conlon & Simpson, 2003; Hayes, 2007; Pelgrum, 2001; Reynolds et al., 2003;Tondeur et al., 2007).

    Thus, after 30 years of research, the main lesson to have been learned is thatintegrating ICT in education is not a technical matter (Earle, 2002; Trucano, 2005;Watson, 2001). ICT should be the tool with which we deliver content and imple-ment educational practices in better ways, based on curriculum and learning princi-ples (Lai, 2008; Marshall & Cox, 2008; Reeves, 2008; Roblyer, 2005; Trucano,2005). Integration is determined not by the quantity or type of ICT used, but byhow and why it is used (Cox & Abbot, 2004; Cox & Webb, 2004; Earle, 2002;Trucano, 2005). Hereafter, when ICT is used to support teaching and learning activ-ities, we refer to it as technology-enhanced instructional design or technology-enhanced learning (TEL) (Dillenbourg, 2008; Kozma, 1991; Roblyer, 2005).

    The available evidence suggests that the use of TEL with specic educationalobjectives, together with an activity that is planned, guided and structured by amediator, is what makes the difference in terms of learning results (Cox & Abbot,

    292 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • 2004; Pilkington, 2008; Trucano, 2005). However, even with effective interventionsbased on TEL, one problem yet to be resolved is their scalability in the educationsystem. This issue is not exclusive to ICT interventions, and the few existing mod-els of scalability identify four dimensions to consider: depth, sustainability, spreadand shift in reform ownership (Coburn, 2003; Dede, 2006; McDonald, Keesler,Kauffman, & Schneider, 2006). Depth refers to the changes needed in classroompractice, while sustainability refers to how to maintain these changes over time andunder what conditions (Batchelor & Norrish, 2005). Spread (or scalability) refers tothe diffusion of the innovation to large numbers of classrooms and schools, i.e., theconditions needed to replicate or scale up the project, and the impact of doing so(Batchelor & Norrish, 2005). The shift in reform ownership refers to the schoolsadoption of the programme.

    Since ICT is just one of many available options for supporting teaching andlearning processes (Earle, 2002), it should have to compete for funding with otheralternatives, in terms of cost-effectiveness or costbenet, regardless of whetherthese others are based on ICT (Harris, 2009; Margolis, Nussbaum, Rodrguez, &Rosas, 2006). To calculate the total cost, an exhaustive inventory must be made ofthe personnel, software, hardware, materials and resources which are needed toimplement a programme. However, such cost parameters are not set out clearly,as is shown by several studies analysing the costbenet of TEL environments(Bates, 2005; Laurillard, 2007; Perraton, Creed, & Robinson, 2002; Rumble, 2001;Steinberg, 2004; Twigg, 2002). Consequently, the cost parameters to be includedand quantied should be specied in operational terms, as well creating a commonlanguage for the educational community (Moyle, 2008).

    Finally, in terms of effectiveness, the study of the impact of ICT includes a widerange of technologies and models whose use is not mainly intended to affectstudent attainment. Therefore, before subsequently evaluating their effects, weshould rst identify which models produce effective instructional practices. Further-more, any such denition must incorporate the topics of ICT integration in schools,scalability and the total cost of ownership, so that we may establish methods forthe systematic design, implementation and evaluation of said programmes.

    1.2. Purpose and structure of this article

    In this article, we use the term educational programme as a specic set of replica-ble procedures, materials and professional development, whose implementationimproves student attainment and that are based on particular resources and/orinstructional processes (Slavin, 2008). Using this concept, we propose a framework the ICT for Education (ICT4E) programme to outline what we understand byan educational programme based on TEL. The objectives of this framework are:

    (1) To dene the elements, concepts, denitions and critical variables involvedin developing a deep understanding of how and why TEL works in realeducational settings.

    (2) To establish the relationships between the expected learning results, theteaching and learning practices supported by TEL, the skills that theserequire, and the activities necessary to develop such skills. This willfacilitate the specication of input, process and outcome indicators for themonitoring and evaluation scheme which must be an integral part of the

    Technology, Pedagogy and Education 293

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • programme (Batchelor & Norrish, 2005; Rodrguez, Nussbaum, Lpez, &Seplveda, 2010; Wagner et al., 2005).

    (3) To dene which elements must be present in an ICT4E programme in terms ofthe processes and resources needed to achieve the desired learning results.

    (4) To calculate the real total cost of an ICT4E programme, allowing it to becompared with other educational programmes in terms of cost-effectiveness.

    Taking the ICT4E programme framework as a base, it is then possible to elabo-rate on the design, implementation and evaluation of its impact.

    The rest of this article is structured as follows: in section 2, we present a frame-work for an ICT4E programme based on the evidence available in the literature,indicating its main components, their interrelation and a brief discussion about thedenition provided. In section 3, we exemplify the framework, using a specic pro-gramme based on computer-supported collaborative learning. In section 4, we dis-cuss the applications of the ICT4E programme that were implemented in Chile. Thearticle nishes with the conclusions and future work in section 5.

    2. Dening an ICT for education programme

    To outline our framework, we undertook a wide review of the literature on ICT ineducation in three categories: curriculum integration, frameworks for the integrationof ICT in schools and impact assessment.

    In the rst category curriculum integration of ICT we found different frame-works that conceptualise the integration of TEL in teaching/learning models (e.g.contents, activities, roles of the teacher and students). For example, curriculum com-ponents (Van den Akker, 2003), domains of educational innovation (Mioduser,Nachmias, Tubin, & Forkosh-Baruch, 2003), learner-centric ecology of resources(Luckin, 2008), and social infrastructure (Bielaczyc, 2006).

    The second category relates to frameworks for the integration of ICT in schools.Here there is an analysis of the factors which affect the implementation of ICT ineducational settings in countries such as Australia (Baskin & Williams, 2006; Hayes,2007), Belgium (Tondeur, Valcke, & van Braak, 2008), Netherlands (Mooij & Sme-ets, 2001), Israel (Nachmias, Mioduser, Cohen, Tubin, & Forkosh-Baruch, 2004;Tubin, Mioduser, Nachmias, & Forkosh-Baruch, 2003), Malaysia (Cloke & Sharif,2001), Great Britain (Kirkman, 2000; Tearle, 2003, 2004) and Singapore (Lim,2002; Lim & Hang, 2003). There are also international studies such as SITESM2(Kozma, 2003; Pelgrum, 2001) and SITES2006 (Plomp, Pelgrum, & Law, 2007).

    The third category corresponds to impact assessments of ICT on education,be this in terms of a state of the art summary of the effect of ICT (Burns & Ungerle-ider, 2003; Cox, 2008; Cox & Abbot, 2004; Cox & Marshall, 2007; Cox & Webb,2004; Lai, 2008; Trucano, 2005), considerations for undertaking monitoring andimpact assessments (Agodini, Dynarski, Honey, & Levin, 2003; Light, 2008; Penuel,2005; Wagner et al., 2005), reviews of programmes and cases (Bakia, Means, Galla-gher, Chen, & Jones, 2009; Balanskat, Blamire, & Kefala, 2006; Culp et al., 2003),meta-analysis research (Kulik, 2003; Liao & Hao, 2008; Waxman, Lin, & Michko,2003), studies based on large-scale research design (Barrera-Osorio & Linden, 2009;Campuzano, Dynarski, Agodini, & Rall, 2009; Dynarski et al., 2007) and researchmethodologies about this topic (Batchelor & Norrish, 2005; Cox, 2008; Marshall &Cox, 2008; Pilkington, 2008; Reeves, 2008).

    294 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Through the second category, we identied three main settings which can beclassied by the aspects of integration of ICT in the school system: classroom,school and external. By classroom, we mean the setting where teaching and/orlearning occurs in the school, even though it might not take place specicallywithin a classroom (e.g., in a computer lab). School denes the specic educa-tional context in which TEL is integrated, while the external setting considersthose processes which are outside of school (e.g. monitoring and evaluation, publicpolicies, costing parameters).

    The classication of the aspects according to the previously dened settings isshown in Table 1. In this table we include the aspects described in the literaturethat, in our opinion, are the most relevant, some of which are transversal across thedifferent settings (A5 to A10).

    Thus, we will dene an ICT4E programme as an educational programme whichintegrates the use of a specic TEL environment into teaching and learning prac-tices. Such TEL environments should modify teaching and learning process in waysthat could not be achieved without ICT support. We will call the model for the useand integration of TEL in the teaching/learning process the pedagogic model.

    We specify an ICT4E programme by identifying the processes and the relation-ship between them in the integration of the pedagogic model in a school. We assignfour attributes to each process: setting, time, aim and outcomes. Setting refers towhere the process operates (i.e., classroom, school or external), while time refers toits duration. The third attribute is the main aim of the process, which can be classi-ed as one of the following:

    (1) Implementation: the use of the pedagogic model by the school team.(2) Intervention: the development of skills and practices in school staff. This is

    normally done by an external team which carries out training, ensuring thegradual adoption of the pedagogic model and accuracy of its implementa-tion, using a monitoring and evaluation scheme.

    (3) Transference: the professional development of the external team so that theintervention can be accurately carried out on a massive scale.

    The fourth attribute is the outcomes, which are the expected skills and/or prac-tices to be developed in the educational actors (i.e., students, teachers, ICT andschool staff). Through the four attributes of the process (setting, time, aim and out-comes), a causal relationship between what happens in the different settings mustbe dened. Therefore, in each process starting with a set of pre-existing skills andpractices amongst the educational actors new skills or practices are developed asoutcomes. For example, in the implementation, the outcomes are the learning resultswhich the ICT4E programme promotes amongst the students. In the intervention,the outcomes are the skills and practices which the teachers need in order to per-form implementation. Finally, in the transference, the outcomes are expressed interms of the skills and practices that the external team requires to carry out aneffective intervention.

    In sections 2.1 to 2.4 we will describe the implementation, intervention, transfer-ence and calculation of total costs, and how these are grounded in the aspects of theliterature identied in Table 1, to which we will refer with the respective numeration.

    Technology, Pedagogy and Education 295

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Table

    1.Classicatio

    nof

    theaspectsinvolved

    inICTin

    educationbasedon

    theliterature,

    accordingto

    thesetting

    inwhich

    they

    occur.

    296 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • 2.1. Implementation

    This is a set of methodological strategies supported by a TEL environment, whoseadoption within a school implies a change or broadening of the existing practices(A5) and the development of new skills amongst the educational actors, with spe-cic roles dened by these strategies (A6). The strategies must consider explicitimprovement goals (A1), dened within an educational context where they operate(A12), based on teaching and learning theories and models (A2) and using specicresources (A3) in a determined conguration (A4). The pedagogic model supportedby TEL is dened by ve elements:

    Purpose (A1): particular subjects, levels and specic curriculum contents. Target schools (A12): schools on which the programme is focused, expressedin terms of geographic surroundings (urban or rural), social vulnerability ofits students and their level of attainment (e.g. measured by local or nationalstandardised tests). Other dening factors could be the type and number ofstudents and school organisation (e.g. multigrade classrooms).

    Implementation resources (A3): all the resources and supporting materialsused by the educational actors (e.g. software, hardware, physical infrastruc-ture, digital and/or non-digital materials, class scripts for teachers and studyguides for students).

    Pedagogic model (A2 to A6): describes how the TEL environment is inte-grated and supports teaching and learning processes. Its main elements are:

    How they learn: how teaching-learning process are modied (e.g. prepara-tion of a class and its development); how the relationships between actors(e.g. teacherstudent, studentstudent) inside and outside the classroom areaffected; with which conguration (e.g. individual, group) they learn; andhow implementation resources are integrated into existing ones.

    Assessment of learning results: how expected effects are evaluated. Who participates within or outside the school (e.g. teachers, classmates,

    parents and family), their roles, and the skills and practices which theyneed to develop.

    Specic didactic context(s): where (e.g., classroom, computer laboratory orhome) and with whom (i.e., classmates, parents, online tutors) they are used.

    These elements can be described using the terminology of existing integrationframeworks, such as curriculum components (Van den Akker, 2003), domainsof educational innovation (Mioduser et al., 2003), learner-centric ecology ofresources (Luckin, 2008), and social infrastructure (Bielaczyc, 2006).

    Implementation outcomes (A1): the expected effects expressed in terms suchas learning results (e.g., scores in standardised tests, learning of speciccurricular contents, acquisition of twenty-rst-century skills), personalisedlearning environments, or data collection for monitoring student progress.

    2.2. Intervention

    So that teachers and students develop the necessary autonomy for sustainable use ofthe pedagogic model (A14) and according to the educational context where they

    Technology, Pedagogy and Education 297

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • take place (A12), the external team will carry out professional development activi-ties (e.g. teacher training, practical experiences, classroom observation) according toa work plan. The outcomes of such activities are the skills and practices needed bythe different educational actors (A8, A11), which are specic to the proposed peda-gogic model, to achieve the learning results set out by the ICT4E (A1). The inter-vention is dened by ve elements:

    Work plan (A8): activities and personnel involved (educational actors and/orexternal team) with the number of person-hours and professional prolerequired according to the design of the intervention.

    Monitoring and evaluation scheme (A9, A13, A14): its objectives are: 1) tomeasure the delity of the implementation; 2) to evaluate the outcomes; and 3)to deliver information for decision making during the intervention, especially soas to take remedial action. The delity of the implementation denes metrics ofwhen and how the new teaching and learning practices should be manifested inthe classroom. This implies a diagnostic of the necessary pre-existing skills andpractices amongst the actors (which take on the programme in each setting), aswell as following the evolution of the programmes adoption over time (A13).This will allow for study of the relationship between the variability of the imple-mentation, the achieved learning results and the adoption by teachers (A14).

    Intervention resources (A3): all the resources and supporting materials(e.g. documentation, software, hardware, observation guidelines and softwaremanuals) used by the external team.

    External team (A15): those who carry out the intervention with their respec-tive role (e.g. a distance tutor or eld professional) and professional prole(e.g. studies, expected or desirable work experience, prior skills).

    Intervention outcomes (A8, A11): skills and practices of the schools educa-tional actors to implement the pedagogic model.

    2.3. Transference

    The transference is the professional development and training of the external teamso that the intervention may be applied autonomously, adapting it according to thecontext of the specic school, following effectiveness standards dened by thedesigners of the ICT4E programmes (transference outcomes). For each of the exter-nal team roles, the previous background must be specied, as well as the new skillsand practices required to perform the role (e.g. how to apply the pedagogic model,carry out intervention activities and apply the assessment instruments from the mon-itoring and evaluation scheme). The transference also includes the evaluation of thedelity of the intervention carried out by the external team (A15).

    2.4. Total cost

    The total cost of the ICT4E programme (A7, A10) refers to the cost of implementa-tion, intervention and transference for the duration of the programme.

    Implementation cost: the sum of the marginal costs which the school incursfor participating in the programme: the person-hours of the educational actors,resources (including software), equipment (including hardware), infrastructure,materials and others (e.g. administration costs).

    298 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Intervention cost: comprises the xed and variable costs of the external team.The variable costs depend on the number of person-hours which the externalteam spends on the intervention (including associated materials, expenses andresources), while the xed costs refer to the general and administrative costs(e.g. management personnel, physical infrastructure) which must be dividedpro-rata according to the number of schools to be serviced and the duration ofthe programme (Margolis et al., 2006).

    Transference cost: the set-up cost of the external team, which is divided pro-rataamongst the participating schools, as are the xed costs of the external team.

    With all of the above, the total cost of the programme is calculated. It can beexpressed in different units such as average cost per school, per classroom and perparticipating student, depending on the comparison to be performed.

    2.5. Discussion of the proposed framework

    In the proposed framework, the processes and settings dened in sections 2.1 to2.4, cover aspects A1 to A15 identied in the literature as relevant (Table 1). TheICT4E programme model is outlined in Figure 1.

    In Figure 1, the block arrows represent the three processes from the ICT4E pro-gramme in its corresponding setting, producing outcomes once each one has beencarried out: learning results in students and the respective skills and practices in theICT and school staff and the external team. Figure 1 also outlines the relationsbetween the main roles of the ICT4E programmes three processes: rstly, the teach-ers perform the implementation. Secondly, members of the external team and theICT and school staff support each other mutually as they carry out the intervention,

    Figure 1. Process involved in the ICT4E programme framework.

    Technology, Pedagogy and Education 299

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • while the external team nally monitors and evaluates the implementation. Next, wewill discuss how our framework meets the objectives set out in section 1.2.

    While the design of the TEL-supported pedagogic model can be carried outtogether with the nal users through design research methodologies (Bielaczyc,2006; Rodrguez, 2010; Van den Akker, Gravemeijer, McKenney, & Nieveen, 2006),it is very unlikely that the schools will design, implement and scale-up educationaltechnologies, because it is not their role. Thus, it is more probable that an ICT4Eprogramme take the form of an externally developed school reform, which typicallyinvolves a short-term inux of resources, making it especially vulnerable to sustain-ability problems, because assistance to facilitate implementation dissipates over time(Coburn, 2003). To address this problem, the ICT4E programme has at its centre thedenition of skills and practices needed by the actors in each process, as well as themonitoring and evaluation of its adoption: from the teachers and the school takingcontrol of the programme, to the external team responsible for the intervention.

    Our framework establishes an order of precedence for the processes, whichstarts with the selection and training of the external team and which concludes withthe measurement of student learning results. However, all of these professionaldevelopment processes require a prior and explicit denition of the activities in theimplementation and intervention. For this reason, the design of an ICT4Eprogramme must be carried out in reverse order, starting with the expected learningresults and the pedagogic model and nishing with the external team.

    With respect to the scalability of the ICT4E programme, the issue of depth iscore to the pedagogic model. The sustainability and shift in reform ownership arethe main aims of the monitoring and evaluation scheme from the interventioncomponent, while the spread is the main concern of the transference component.The ICT4E programme assigns the responsibilities of these last two dimensions tothe external team, which involves designing strategies to achieve these goals rightfrom the design of the ICT4E programme.

    Regarding the costing parameters, as well as being consistent with those previ-ously identied in studies of the costs of TEL (Bates, 2005; Laurillard, 2007; Perr-aton et al., 2002; Rumble, 2001; Steinberg, 2004; Twigg, 2002), our frameworkincorporates another two which have not previously been identied: the cost associ-ated with monitoring and evaluation, and the cost of the transference. Our frame-work does not consider the programmes design costs because these are consideredas sunk costs. We also assume that the evaluation of the design and effectivenesshave already been resolved. This topic is discussed extensively by Rodriguez, Nuss-baum, and Dombrovskaia (2011) through the Evolutionary Development model forICT4E programmes.

    3. Describing a specic ICT4E programme: the Eduinnova case

    In this section we will illustrate how the proposed framework (Figure 1) allows usto specify the intervention and transference by identifying the skills and practicesrequired by the actors involved for the implementation of the pedagogic model, asestablished by the ICT4E programme. This will be exemplied with Eduinnova, anICT4E programme which uses mobile technology inside the classroom to supportcollaborative learning with mobile devices such as laptops, netbooks or PersonalDigital Assistants (PDAs). These devices are used with a one-to-one computingmodel (Penuel, 2006), but shared by various classrooms throughout the day.

    300 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • 3.1. Implementation

    The purpose of Eduinnova is to support student attainment through computer-sup-ported collaborative learning (CSCL) (Dillenbourg, 1999), constructivist (Zurita &Nussbaum, 2004a) and collaborative learning approaches (Zurita & Nussbaum,2004b). Currently, the programme covers the contents of the curriculum for mathe-matics and Spanish language (third to eighth grade), science and social studies(third and fourth grade) in elementary education; and for mathematics, Spanish lan-guage, physics, chemistry, biology, history and geography in secondary education.

    The target schools are elementary and/or secondary urban state-subsidisedschools, from lower-middle socioeconomic strata. In terms of infrastructure, to berun adequately, Eduinnova requires classrooms with at least one power outlet andteacher access to a computer with a broadband connection.

    The implementation resources are: the Eduinnova TEL environment (softwarefor laptop or PDA), a laptop (or PDA) equipped with wireless communication (Wi-Fi) for each student and teacher; a wireless router which can support at least 45connections to be used in the classroom; and a cabinet for storing, transporting andcharging the devices batteries. In addition, an online library is provided, that: 1)contains CSCL activities previously located by experts within a curriculum frame-work, but customisable by teachers in schools; 2) registers the teachers use of thetechnology; and 3) stores the results of the activities undertaken in class. All ofthese resources are used to complement the traditional classroom materials (e.g.books and concrete material).

    Next, we will describe the elements of the pedagogic model of Eduinnova fornetbooks/PDAs, to be implemented in a school. Firstly, the way in which the stu-dents learn is outlined in Figure 2.

    The teacher accesses an online library in which they can search, create anddownload CSCL activities for their class (Figure 2, step a). The PDAs/netbooksare taken to the classroom in the storage cabinet. In the classroom, using theEduinnova TEL environment, the teacher chooses and sends the CSCL activityto their students (Figure 3, step b) and the class is randomly divided into groupsof three (Figure 2, step c). In each group, the students must solve a problemtogether, with the aid of traditional materials (e.g. books, notes), while theirinteraction is mediated by the TEL environment (Figure 3a). In addition, thesoftware also gives them feedback on the answer and does not allow the stu-dents to progress until they arrive at the solution to the given problem (Cortezet al., 2005; Nussbaum et al., 2009; Rodrguez et al., 2010). At the same time,the TEL environment delivers reports to the teacher in real time about the per-formance of the students, giving them the opportunity to mediate the work doneby each group (Figure 2, step d and Figure 3b). This information is also backedup on the web for later use (Figure 2, step e).

    Secondly, in the pedagogic model, the assessment of the learning results consid-ers the formative and summative stages. In the formative stage, the teacher givesfeedback to specic groups or to the whole class during the collaborative work. Inthe summative assessment stage, the students answer online individual tests in real-time, supported by the TEL environment. The teacher can also evaluate studentlearning by administering their own tests.

    Thirdly, the roles, skills and practices and educational actors that participate inthe implementation are shown in Table 2.

    Technology, Pedagogy and Education 301

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • To nish the description of the pedagogic model, the classroom is the specicdidactic context where this TEL environment is used, with the students from thatclass and during the school timetable, for the specic subject in which it is applied.

    The implementation outcome, which is expected to be achieved following theadoption of the programme, is an effect size above 0.25 in the subjects in which itis applied. This is according to the evidence available on the impact of ICT onlearning (Agodini et al., 2003; Dynarski et al., 2007; Liao & Hao, 2008).

    Figure 3. Snapshots of a class which is supported by the collaborative teaching methodEduinnova: (a) general panorama of the classroom, (b) students work together with the helpof the teacher.

    Figure 2. Using Eduinnova TEL environment in school.

    302 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Table2.

    Roles,skillsandpractices

    oftheeducationalactors

    intheEduinnova

    pedagogicmodel.

    Who

    Roles

    Skills&

    practices

    Students

    Collaborativ

    elearning,basedon

    4factors(Cortezet

    al.,2005):

    Individual

    responsibility(S1).

    Mutualsupport(S2).

    Positive

    interdependence(S3).

    Individual

    responsibility:

    each

    studentisin

    charge

    oftheirow

    nworkand

    effortto

    learn.

    Socialface-to-face

    interaction(S4).

    Mutualsupport:teaching

    theothermem

    bers

    ofthegroup.

    Content

    learning

    inspecicsubject(S5).

    Positive

    interdependence:

    collaboratio

    nisonly

    successful

    whenevery

    studenthasinteracted

    andaccomplishedtheirindividual

    goalsto

    solvethe

    groupproblem.

    Socialface-to-face

    interaction:

    thedecision

    makingprocessrelieson

    the

    ability

    toefciently

    exchange

    opinions,negotiate

    andconstructan

    answ

    ertogether.

    Teachers

    From

    expert'to

    facilitator,mediatorandguide(Condie&

    Munro,2007),

    supportin

    g,monito

    ring

    andprovidingfeedback

    whenstudentswork

    collaborativ

    ely.

    Technologicalhandlin

    gof

    theEduinnova

    TEL(S6).

    Mediate

    students'group

    workinside

    the

    classroom

    andmonito

    rtheiroutcom

    es(S7).

    Integratemobile

    CSCLactiv

    ities

    into

    alecturein

    accordance

    with

    the

    subject,objectives,tim

    ingandresources(digitalor

    non-digital)involved.

    Givecontinuous

    feedback

    basedon

    theresults

    obtained

    (S8).

    Createnew

    activ

    ities,select

    orcustom

    iseexistin

    gones

    from

    theonlin

    elib

    rary.

    IntegrateCSCLmethodology

    into

    theirow

    nteaching

    styles

    andotheractiv

    ities

    and

    resourcesfrom

    theirsubject(S9).

    Sharesmaterials,know

    ledgeandexperience

    ofcollaborativ

    eteaching

    practices

    with

    otherteachers.

    Adapt

    digitalresourcesfrom

    theprogrammeto

    suittheirow

    nneeds(S10).

    (Con

    tinued)

    Technology, Pedagogy and Education 303

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Table

    2.(Con

    tinued).

    Who

    Roles

    Skills&

    practices

    ICT&

    school

    staff

    ICTstaff:transport,recharging

    devicesandsolvingtechnicalissues

    inside

    theclassroom.

    Technicalsupportfortheteacher(S11).

    Schoolstaff:planning

    andmonito

    ring

    thecurriculum

    andprogress

    ofthe

    students.

    Managethedevicesin

    orderto

    guaranteeand

    maxim

    isetheiraccess

    anduse(S12).

    Carry

    outmonito

    ring

    controlandpedagogical

    supportto

    teacherwhenusingthe

    programme(S13).

    Plantheuseof

    technology,consistent

    with

    institu

    tionalinterestsandthepedagogical

    needswhich

    aredetected

    (S14).

    304 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • In sections 3.2 to 3.4, we will describe how skills and practices are developedduring the intervention and transference, as well as how its adoption is monitoredand evaluated.

    3.2. Intervention

    The objectives of the intervention in Eduinnova are: 1) to develop the skills andpractices S1 to S14; 2) to measure its adoption through the systematic applicationof assessment instruments which form part of the monitoring and evaluationscheme; and 3) to take remedial actions in case of excessively large gaps in relationto the expected evolution.

    Eduinnova shows that, although the ICT4E programme framework sets an orderof precedence between implementation and intervention, these do not have to bestrictly sequential. The intervention in Eduinnova consists of an implementation withcoaching inside the classroom which looks to develop a hands-on experienceamongst the teachers, with a formative and summative evaluation of their activities,in order to develop skills and practices S6 to S10. Thus, the implementation andintervention overlap to support the teachers in the integration of TEL in their classes.

    The work plan of the intervention takes place over two years and is divided intofour main phases and 12 activities, as detailed in Table 3, which also indicates theskills and practices to be developed during each phase.

    The key phase of the plan is Phase 2 (Training and Coaching), where theteachers learn to use the TEL environment, both pedagogical and technological, tocarry out the practices of the programme (S6 to S10). It is expected that in thisphase the pedagogic model will be adopted, veried by a participant observation ofthe teachers in their rst class using the TEL environment (activity 5). In this obser-vation, any doubts are cleared up and the work done during the face-to-face trainingis reinforced (activity 4). According to the results of this observation, additionalsupport is given to those teachers that have adopted the pedagogic model to a lesserextent. Subsequently, the level of adoption is only measured in the day-to-dayactivity of the teachers with non-participant observation, giving them individualfeedback in writing and detailing their main strengths as well as giving specic sug-gestions (activity 6). When necessary, remedial actions are taken (activity 9). Theother activities from Phase 2 are: giving technical support to schools (activity 7)and monitoring and evaluating the programme (activity 8). Skills and practices S11to S14 are acquired by working with other educational actors within the school.

    The monitoring and evaluation scheme has four adoption indicators: teacherstechnical skills (S6), pedagogical skills in collaborative learning (S7 and S8), ICTintegration within the curriculum (S9 and S10) and ICT management inside theclassroom (S6 plus efciency of use of classroom time). The assessment instru-ments are observation protocols based on previously validated rubrics. Studiesindicate that if these indicators do not reach a sufcient nal threshold, statisticallysignicant differences are produced in the learning amongst students whose teachershave different levels of adoption (Rodrguez et al., 2010). For further details anddiscussion of this monitoring and evaluation scheme, please refer to Rodriguezet al. (2010).

    The most important intervention resources are the assessment instruments of theadoption of the practices, for example, the previously mentioned observation proto-cols (activity 5).

    Technology, Pedagogy and Education 305

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Table3.

    Workplan

    forEduinnova

    programme(Rodrguez

    etal.,2010).

    306 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • The external team performs, mediates and supports the intervention. The rolesare: project-coordinator professionals who coach teachers inside the classroom(pedagogical assistants) and technicians who support school ICT staff (technicalassistants). The professional proles of each are the following:

    Project coordinator: educational professional (e.g. teacher or educational psy-chologist) with more than three years of experience and ICT skills at complextools level, as established in UNESCOs (2008) ICT competency frameworkfor teachers.

    Pedagogical assistants: teacher with more than two years of experience andknowledge of the internal organisation of the schools according to the countrywhere the programme is implemented, and ICT skills at complex tools level.

    Technical assistants: programmer or systems analyst.

    In the case of classroom and technical assistants, each assistant works with acertain number of schools depending on the phase of intervention in which eachschool can be found. Thus, the organisational structure of the external team dependson the number of schools participating in the programme.

    The intervention outcomes are the skills and practices S1 to S14, dened insection 3.1 (Table 2).

    3.3. Transference

    The transference is the professional development and training of the external teamin three areas: implementation, intervention and use of technology applied in theprogramme at advanced level. They are also trained in the application of assessmentinstruments from the monitoring and evaluation scheme, so as to guarantee thequality and reproducibility of the programmes implementation.

    3.4. Total cost

    The calculation of the total cost is exemplied by calculating the intervention cost.For this, we will take activity 5 (Coaching teachers) mentioned in section 3.2(Table 3). Activity 5 is divided into two activities. In the rst (5.1 Observation ofeach teachers rst class) the teachers work is observed and supported, reinforcingpreviously acquired skills and following an observation guideline after the class. Inthe second activity (5.2 Teacher observation) the teachers that show most signsof difculty are further supported. If each observation lasts approximately 1.5 hours,and assuming that a maximum of 15 teachers are used in each school with an aver-age rate of 20% of teachers showing a low level of adoption (of previous experi-ences), 22.5 and 4.5 person-hours are used for activities 5.1 and 5.2, respectively.Considering in Chile a cost of US$9/person-hour (pedagogical supporter prole),material costs (US$7) and transport (US$10), we arrive at a total cost of US$260per school for this activity, given that it is only carried out during the rst year (seeTable 3).

    Similarly, the variable costs for the remaining activities are calculated. The xedcosts of the intervention and transference are calculated according to the scale ofthe schools where it is to be implemented.

    Technology, Pedagogy and Education 307

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • 4. Current applications of ICT4E programme

    The ICT4E programme framework is used as a specication for the ICT curriculumintegration strategies supported by Enlaces (http://www.enlaces.cl), the educationaltechnology agency from the Chilean Ministry of Education (Rodrguez, 2008). Allprogrammes that apply for funding from said agency must adjust themselves to thisframework. To determine if a programme ts the specication, an evaluation standardis applied (Romn, 2010). This standard uses the Context Input Processes Products(CIPP) methodology of analysis (Stufebeam & Shinkeld, 2007) to dene the indica-tors for the quality of the design, processes and impact of an ICT4E programme.

    When a programme passes the evaluation, it is included in the catalogue of digi-tal educational resources promoted by Enlaces for all schools in Chile (http://www.catalogored.cl). In addition to Eduinnova, there are other ICT4E programmescurrently included in said catalogue, such as those for one-to-one learning (Lagoset al., 2009; Lpez, Parada, & Gmez, 2009), project-based learning (Snchez &Alarcn, 2003), mathematics (Daz, 2010), Spanish language (Basez, Bastas,Villalobos, & Cafena, 2010), science (Miranda & Feris, 2009; Moenne, 2008, 2010)and civic education (Contreras, 2008).

    Additionally, the ICT4E programme framework allows for a more accurate esti-mation of the total cost of integrating ICT, because it helps to determine all of thematerials and resources which the programme needs. Such estimations enable ananalysis of the economic feasibility of the programme, determining whether or not itcan be implemented given the nancial restrictions on school and Ministry of Educa-tion spending (Margolis et al., 2006). Another possible application is to compareprogrammes using a cost-effectiveness analysis (Belli, Anderson, Barnum, Dixon, &Tan, 2001; Creemers & van der Werf, 2000; Feinstein & Picciotto, 2000; Harris,2009; Margolis et al., 2006). The ICT4E programme model offers a cost-comparisonframework, standardising the process components and resource prices of the pro-grammes under comparison. For example, the denition of the proles for the exter-nal team facilitates the comparison between programmes as professionals withsimilar training and experience should have similar costs within the same country.

    The ICT4E programme supported the design of the Eduinnova programme,which, in the last decade, has been tested in different grade levels and with differentsubjects: preschool education (Infante et al., 2009), elementary schools (Zurita &Nussbaum, 2004b), secondary schools (Rodrguez et al., 2010) and university level(Bustos & Nussbaum, 2009; Valdivia & Nussbaum, 2007, 2009). The transferencecomponent allowed Eduinnova to also be implemented in schools in Argentina,Brazil, Guatemala, Great Britain (Nussbaum et al., 2009) and the United States(Roschelle et al., 2010). The effect sizes found in quasi-experimental studies uctu-ate between 0.32 and 0.41 (Rodrguez et al., 2010, 2011), consistent with theexpected learning results for an educational technology (section 3.1).

    Finally, another nonetheless important application was the development of amodel for the design, implementation and evaluation of ICT4E programmes. Saidmodel called Evolutionary Development builds each component of the ICT4Eprogramme in stages and validates it in real educational settings through formativeand summative evaluations based on design research methodologies and usingexperimental or quasi-experimental studies (Rodrguez et al., 2011). Thus, theefcacy of the implementation, the effectiveness of intervention and the efciencyof the programme as a whole are tested empirically.

    308 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • 5. Conclusions and future work

    In this article, we proposed the ICT4E programme framework to respond to thequestion about what an educational programme supported by ICT should look like(section 2). We centred the framework on producing a sustainable change in timethrough the effective adoption of new teaching and learning practices supportedby TEL environments. The key components of an ICT4E programme are: theimplementation of the TEL-supported pedagogic model, intervention, transferenceand total costs. The transference component comprises the formation strategies ofan external team to guarantee the replicability and quality of the intervention.

    The proposed framework (objective 1 of the article) makes explicit the relation-ships between each of the activities and the expected outcomes (at an intermediateand nal level) in each process of the ICT4E programme (objective 2). Secondly, itallows educators to identify which elements (e.g. skills and practices, actorsinvolved, activities, resources) can be absent (objective 3). Finally, it enables thecalculation of the real total cost of the programme (objective 4). These featureswere illustrated by the Eduinnova programme in section 3.

    Currently, in Chile, the ICT4E programme model is applied to specify ICT pro-jects in education subject to public funding and evaluate them according to a stan-dard built using the framework as a base (Romn, 2010). Other applications are theanalysis of economic feasibility and cost-effectiveness (Margolis et al., 2006). Thislast case presents practical challenges as the effectiveness must be measured undercriteria from rigorous evidence (Slavin, 2008). Furthermore, only little is knownabout how the effectiveness and costs of an ICT4E programme vary with time,especially with respect to long-term substitution of obsolete technology.

    Thus, a future challenge for policy making is to determine how the cost-effec-tiveness varies in the medium and long term, and how such changes inuence thecurrent comparison between programmes. To resolve this problem, some authorspropose the calculation of a present value both of the effect size as well as of thecosts, which would imply making suppositions with respect to the discount rateto be applied and the time horizon for the analysis (Harris, 2009). One empiricalsolution would be to carry out longitudinal studies of the population affected byICT4E programmes to see if the effects are sustainable, and determine the rate atwhich they decline in the long term. Periodical cross-sectional studies would alsoverify whether or not an ICT4E programme systematically produces the expectedoutcomes. If it does not, the ICT4E programme model can conduct an audit todiscover which aspects are loss-making and to what extent.

    Another form of comparing educational programmes is through costbenetstudies, which are harder to carry out because the benets must be monetarilyquantied (Belli et al., 2001; Margolis et al., 2006). However, Laurillard (2007)proposed a prospective method which allows the identication of specic benetsof a TEL environment. Our ICT4E programme supports the implementation of saidmethod, as it facilitates the calculation of the costs and also species the intermedi-ary and nal outcomes which should be achieved. Thus, after valuing the identiedcosts and benets, it would be possible to calculate the net present value and returnof investment of implementing an ICT4E programme in a school or district. Withthese indexes policy makers have more quantitative information with which to com-pare other educational alternatives (with or without the technology) being availableto them at the time of evaluating.

    Technology, Pedagogy and Education 309

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • The specication of processes and calculation of the costs that the ICT4Eprogramme enables also allows the application of process improvement and cost-reduction models such as Capability Maturity Model Integration (Forrester, Buteau,& Shrum, 2011) in order to improve the cost-effectiveness of the programme.

    Another aspect to be determined is whether or not there exist signicant differ-ences in the impact of ICT4E programmes which adjust more to the framework, incomparison to those that do it to a lesser degree, a topic which is the subject ofresearch currently being conducted.

    Finally, there are at least two issues for further exploration. Firstly, the proposedframework could be extended to include other aspects of ICT in education bysubstituting the pedagogic model with one designed to support such processes (e.g.,school administration and learning management models). Secondly, our ICT4E pro-gramme framework has the potential to specify less teacher-centred models, likelifelong learning or distance education, by redening the concept of settingswithin the framework. For example, the classroom setting could be extended toother places outside the school such as the home or ofce.

    AcknowledgementsThis research was funded by the Chilean Ministry of Education through the Enlaces Centerfor Educational Technology, and CONICYT under grants FONDECYT 11110424, CIE-05CIAE and CIE-01 CEPPE. Special thanks go to Claudia Morchio and Goryet Pandorfa fromEnlaces, and Eugenio Severin from the Inter-American Development Bank. This paper isdedicated to the memory of Dr Gerardo Moenne, from the Instituto de InformticaEducativa, Universidad de la Frontera, Temuco, Chile.

    Notes on contributorsPatricio Rodriguez is research associate at the Center of Advanced Research in Education ofthe University of Chile. He has more than 15 years experience of educational technology ineducation, with projects implemented in Argentina, Brazil, Chile, United Kingdom andUnited States.

    Miguel Nussbaum is full professor for Computer Science at the School of Engineering ofthe Universidad Catlica de Chile. He carries out research in instructional design based ontechnology.

    Lioubov Dombrovskaia is assistant professor at the Department of Informatics of theUniversidad Tcnica Federico Santa Maria in Santiago, Chile. She teaches and carries outresearch in humancomputer interaction, software design, bioinformatics, engineering educationand technological innovation.

    ReferencesAgodini, R., Dynarski, M., Honey, M., & Levin, D. (2003). The effectiveness of educational

    technology: Issues and recommendations for the national study. Mathematica PolicyResearch. Retrieved from http://www.mathematica-mpr.com/PDFs/edtechrec.pdf

    Bakia, M., Means, B., Gallagher, L., Chen, E., & Jones, K. (2009). Evaluation of theEnhancing Education Through Technology program: Final report. US Department ofEducation, Ofce of Planning, Evaluation and Policy Development, Policy and ProgramStudies Service. Washington, DC. Retrieved from http://www2.ed.gov/rschstat/eval/tech/netts/nalreport.pdf

    Balanskat, A., Blamire, R., & Kefala, S. (2006). The ICT impact report: A review of studiesof ICT impact on schools in Europe. Retrieved from European Commission website:http://ec.europa.eu/education/pdf/doc254_en.pdf

    310 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Barrera-Osorio, F., & Linden, L. (2009). The use and misuse of computers in education: Evi-dence from a randomized controlled trial of a language arts program. Retrieved fromPoverty Action Lab website: http://www.povertyactionlab.org/evaluation/use-and-misuse-computers-education-evidence-randomized-controlled-trial-language-arts-pro

    Basez, C., Bastas, V., Villalobos, R., & Cafena, D. (2010). Piloto @escribo, usando lasTIC para potenciar la escritura: Informe nal [@escribo, using ICT to improve writing.Final report]. Retrieved from Enlaces website: http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3709

    Baskin, C., & Williams, M. (2006). ICT integration in schools: Where are we now and whatcomes next? Australasian Journal of Educational Technology, 22, 455473.

    Batchelor, S., & Norrish, P. (2005). Framework for the assessment of ICT pilot projects.Retrieved from infoDev website: http://www.infodev.org/en/Publication.4.html

    Bates, A. (2005). Technology, e-learning and distance education (2nd ed.). London: Routledge.Belli, P., Anderson, J.R., Barnum, H.N., Dixon, J.A., & Tan, J.-P. (2001). Economic

    analysis of investment operations. Analytical tools and practical applications. Wash-ington, DC: The International Bank for Reconstruction and Development/The WorldBank.

    Bielaczyc, K. (2006). Designing social infrastructure: Critical issues in creating learningenvironments with technology. Journal of the Learning Sciences, 15, 301329.

    Burns, T.C., & Ungerleider, C.S. (2003). Information and communication technologies inelementary and secondary education. International Journal Educational Policy, Researchand Practice, 3(4), 2754.

    Bustos, H., & Nussbaum, M. (2009). An experimental study of the inclusion of technologyin higher education. Computer Applications in Engineering Education, 17, 100107.

    Callister, T.A. (1992). The computer as doorstep: Technology as disempowerment. Phi DeltaKappan, 74, 324329.

    Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveness of reading andmathematics software products: Findings from two student cohorts. Retrieved from Insti-tute of Education Sciences website: http://ies.ed.gov/ncee/pubs/20094041/

    Cartwright, V., & Hammond, M. (2007). Fitting it in: A study exploring ICT use in a UKprimary school. Australasian Journal of Educational Technology, 23, 390407.

    Castro, C. (2004). Are new technologies better technologies? For whom? In D.W. Chap-man & L.O. Mhlck (Eds.), Adapting technology for school improvement: A globalperspective (pp. 3954). Paris: International Institute for Educational Planning,UNESCO.

    Cloke, C., & Sharif, S. (2001). Why use information and communications technology? Sometheoretical and practical issues. Technology, Pedagogy and Education, 10, 718.

    Coburn, C.E. (2003). Rethinking scale: Moving beyond numbers to deep and lasting change.Educational Researcher, 32(6), 312.

    Condie, R., & Munro, B. (2007). The impact of ICT in schools: A landscape review. Coven-try: Becta.

    Conlon, T., & Simpson, M. (2003). Silicon Valley versus Silicon Glen: The impact of com-puters upon teaching and learning: A comparative study. British Journal of EducationalTechnology, 34, 137150.

    Contreras, D. (2008). Civit@s.net, alfabetizacin digital para ciudadanos en red: Informe nal[Civit@s.net, digital literacy for networked citizens. Final report]. Retrieved from Enlaceswebsite: http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=2296

    Cortez, C., Nussbaum, M., Lpez, X., Rodrguez, P., Santelices, R., Rosas, R., et al. (2005).Teachers support with ad-hoc collaborative networks. Journal of Computer AssistedLearning, 21, 171180.

    Cox, M. (2008). Research IT in education. In J. Voogt & G. Knezek (Eds.), Internationalhandbook of information technology in primary and secondary education (pp. 965981).New York, NY: Springer.

    Cox, M., & Abbot, C. (2004). ICT and attainment: A review of the research literature. ICTin Schools Research and Evaluation Series No. 17, Department for Education andSkills and Becta.

    Cox, M., & Marshall, G. (2007). Effects of ICT: Do we know what we should know? Edu-cation and Information Technologies, 12(2), 5970.

    Technology, Pedagogy and Education 311

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Cox, M., & Webb, M. (2004). ICT and pedagogy: A review of the research literature. ICTin Schools Research and Evaluation Series No. 18, Department for Education andSkills and Becta.

    Creemers, B., & van der Werf, G. (2000). Economic viewpoints in educational effectiveness:Cost-effectiveness analysis of an educational improvement project. School Effectivenessand School Improvement, 11, 361384.

    Cuban, L. (1986). Teachers and machines: The classroom use of technology since 1920.New York, NY: Teachers College Press, Columbia University.

    Culp, K.M., Honey, M., & Mandinach, E. (2003). A retrospective on twenty years of educa-tion technology policy. US Department of Education, Ofce of Educational Technology.Retrieved from http://www.ed.gov/rschstat/eval/tech/20years.pdf

    Dede, C. (2006). Scaling up: Evolving innovations beyond ideal settings to challenging con-texts of practice. In R.K. Sawyer (Ed.) The Cambridge Handbook of Learning Sciences(pp. 551566). Cambridge: Cambridge University Press.

    Daz, W. (2010). Proyecto Enlaces Matemtica: Informe nal. Retrieved from http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=4011

    Dillenbourg, P. (1999). Collaborative learning: Cognitive and computational approaches.Advances in learning and instruction series. Elsevier Science, New York.

    Dillenbourg, P. (2008). Integrating technologies into educational ecosystems. DistanceEducation, 29, 127140.

    Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., et al.(2007). Effectiveness of reading and mathematics software products: Findings from therst student cohort. US Department of Education, Institute of Education Sciences.Retrieved from http://ies.ed.gov/ncee/pubs/20074005/

    Earle, R.S. (2002). The integration of instructional technology into public education: Prom-ises and challenges. Educational Technology, 42, 513.

    Feinstein, O.N., & Picciotto, R. (2000). Evaluation and poverty reduction. Proceedings from aWorld Bank conference. World Bank Series on Evaluation and Development (Vol. 3). Wash-ington, DC: The International Bank for Reconstruction and Development/The World Bank.

    Forrester, E., Buteau, B., & Shrum, S. (2011). CMMI for services: Guidelines for superiorservice (2nd ed.). Boston, MA: Addison-Wesley.

    Harris, D.N. (2009). Toward policy-relevant benchmarks for interpreting effect sizes: Com-bining effects with costs. Educational Evaluation and Policy Analysis, 31, 329.

    Hayes, D.N.A. (2007). ICT and learning: Lessons from Australian classrooms. Computers &Education, 49, 385395.

    Hennessy, S., Ruthven, K., & Brindley, S. (2005). Teacher perspectives on integrating ICTinto subject teaching: Commitment, constraints, caution, and change. Journal of Curricu-lum Studies, 37, 155192.

    Infante, C., Weitz, J., Reyes, T., Nussbaum, M., Gmez, F., & Radovic, D. (2010). Co-located collaborative learning video game with single display groupware. InteractiveLearning Environments, 18, 2, 177195.

    Kikis, K., Scheuermann, F., & Villalba, E. (2009). A framework for understanding and eval-uating the impact of information and communication technologies in education. In F.Scheuermann & F. Pedr (Eds.), Assessing the effects of ICT in education: Indicators,criteria and benchmarks for international comparisons (pp. 6982). Luxembourg: Publi-cations Ofce of the European Union.

    Kirkman, C. (2000). A model for the effective management of information and communica-tions technology development in schools derived from six contrasting case studies. Tech-nology, Pedagogy and Education, 9, 3752.

    Kozma, R.B. (1991). Learning with media. Review of Educational Research, 61, 179211.Kozma, R.B. (2003). Technology and classroom practices: An international study. Journal of

    Research on Technology in Education, 36, 114.Kulik, J. (2003). Effects of using instructional technology in elementary and secondary

    schools: What controlled evaluation studies say. Technical report, SRI International.Lagos, M.E., Vil, G., Miranda, H., Contreras, D., Rodrguez, J., & Sandoval, K. (2009).

    Aprendiendo Matemtica y Lenguaje con Tecnologa Porttil 1:1: Informe nal [Learn-ing mathematics and language using 1:1 computing. Final report]. Retrieved from Enlac-es website: http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3626

    312 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Lai, K.-W. (2008). ICT supporting the learning process: The premise, reality, and promise.In J. Voogt & G. Knezek (Eds.), International handbook of information technology inprimary and secondary education (pp. 215230). New York, NY: Springer.

    Laurillard, D. (2007). Modelling benets-oriented costs for technology enhanced learning.Higher Education, 54, 2139.

    Liao, Y.C., & Hao, Y. (2008). Large-scale studies and quantitative methods. In J. Voogt &G. Knezek (Eds.), International handbook of information technology in primary and sec-ondary education (pp. 10191035). New York, NY: Springer.

    Light, D. (2008). Evaluating educational technology interventions: How do we know it isworking? Center for Children and Technology, Education Development Center. Retrievedfrom http://cct.edc.org/report.asp?id=254

    Lim, C.P. (2002). A theoretical framework for the study of ICT in schools: A proposal. Brit-ish Journal of Educational Technology, 33, 411421.

    Lim, C.P., & Hang, D. (2003). An activity theory approach to research of ICT integration inSingapore schools. Computers & Education, 41, 4963.

    Lpez, M., Parada, D., & Gmez, F. (2009). MIC (1:1) Modelo de Integracin Curricularcon estrategia 1 a 1. Resultados de la Implementacin: Informe nal [Curricular integra-tion model with 1:1 computing. Final results]. Retrieved from Enlaces website: http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3677

    Luckin, R. (2008). The learner centric ecology of resources: A framework for using technol-ogy to scaffold learning. Computers & Education, 50, 449462.

    Margolis, J., Nussbaum, M., Rodrguez, P., & Rosas, R. (2006). Methodology for evaluatinga novel education technology: A case study of handheld video games in Chile. Comput-ers & Education, 46, 174191.

    Marshall, G., & Cox, M. (2008). Research methods: Their design, applicability and reliabil-ity. In J. Voogt & G. Knezek (Eds.), International handbook of information technologyin primary and secondary education (pp. 9831002). New York, NY: Springer.

    McDonald, S.-K., Keesler, V.A., Kauffman, N.J., & Schneider, B. (2006). Scaling-up exem-plary interventions. Educational Researcher, 35(3), 1524.

    McFarlane, A. (2001). Perspectives on the relationships between ICT and assessment. Jour-nal of Computer Assisted Learning, 17, 227234.

    Mioduser, D., Nachmias, R., Tubin, D., & Forkosh-Baruch, A. (2003). Analysis schema forthe study of domains and levels of pedagogical innovation in schools using ICT. Educa-tion and Information Technologies, 8, 2336.

    Miranda, S., & Feris, C. (2009). Modelo de Informtica Educativa Aprendizaje Interactivode las Ciencias. Enseanza Media: Informe Final [Interactive learning of science forsecondary students. Final report]. Retrieved from Enlaces website: http://intranet.reden-laces.cl/index.php?id=11414&no_cache=1&uid=3740

    Moenne, G. (2008). Dibujos con Vida. Informe nal [Drawings alive. Final report].Retrieved from Enlaces website: http://intranet.redenlaces.cl/index.php?id=11414&no_ca-che=1&uid=2186

    Moenne, G. (2010). Atenea: Desarrollo del pensamiento cientco a travs de las TIC: Informenal [Development of scientic thinking using ICT. Final report]. Retrieved from Enlaceswebsite: http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=3839

    Mooij, T., & Smeets, E. (2001). Modelling and supporting ICT implementation in secondaryschools. Computers & Education, 36, 265281.

    Moonen, J. (2008). Policy from a global perspective. In J. Voogt & G. Knezek (Eds.),International handbook of information technology in primary and secondary education(pp. 11711178). New York, NY: Springer.

    Moyle, K. (2008). Total cost of ownership and total value of ownership. In J. Voogt & G.Knezek (Eds.), International handbook of information technology in primary and sec-ondary education (pp. 615631). New York, NY: Springer.

    Nachmias, R., Mioduser, D., Cohen, A., Tubin, D., & Forkosh-Baruch, A. (2004). Factorsinvolved in the implementation of pedagogical innovations using technology. Educationand Information Technologies, 9, 291308.

    Nussbaum, M., Alvarez, C., McFarlane, A., Gomez, F., Claro, S., & Radovic, D. (2009).Technology as small group face-to-face collaborative scaffolding. Computers & Educa-tion, 52, 147153.

    Technology, Pedagogy and Education 313

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Pelgrum, W. (2001). Obstacles to the integration of ICT in education: Results from a world-wide educational assessment. Computers & Education, 37, 163178.

    Penuel, W.R. (2005). Recommendations for evaluation research within educational innova-tions. Center for Technology in learning, SRI International. Retrieved from http://ctl.sri.com/publications/downloads/Rec_Eval_Research_within_Edu_Innovations.pdf

    Penuel, W.R. (2006). Implementation and effects of one-to-one computing initiatives: Aresearch synthesis. Journal of Research on Technology in Education, 38, 329348.

    Perraton, H., Creed, C., & Robinson, B. (2002). Teacher education guidelines: Using openand distance learning. Technology, curriculum, cost, evaluation. UNESCO, Higher Edu-cation Division, Teacher Education Section.

    Pilkington, R.M. (2008). Measuring the impact of information technology on students learn-ing. In J. Voogt & G. Knezek (Eds.), International handbook of information technologyin primary and secondary education (pp. 10031018). New York, NY: Springer.

    Plomp, T., Pelgrum, W., & Law, N. (2007). SITES2006 International comparative surveyof pedagogical practices and ICT in education. Education and Information Technologies,12, 8392.

    Reeves, T. (2008). Evaluation of the design and development of IT tools in education. In J.Voogt & G. Knezek (Eds.), International handbook of information technology in primaryand secondary education (pp. 10371051). New York, NY: Springer.

    Reiser, R.A. (2001). A history of instructional design and technology: Part I: A history ofinstructional media. Educational Technology Research and Development, 49, 5364.

    Reynolds, D., Treharne, D., & Tripp, H. (2003). ICT The hopes and the reality. BritishJournal of Educational Technology, 34, 151167.

    Robertson, J. (2002). The ambiguous embrace. Twenty years of IT (ICT) in UK primaryschools. British Journal of Educational Technology, 33, 403409.

    Roblyer, M.D. (2005). Educational technology research that makes a difference. Series intro-duction. Contemporary Issues in Technology and Teacher Education, 5, 192201.

    Rodrguez, P. (2008). Estndar de Modelo de Informtica Educativa [Standard for an ICT foreducation programme]. Enlaces, Center for Educational Technology. Ministry of Education.Retrieved from http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=2549

    Rodrguez, P. (2010). A methodology for replication, scaling-up and transference of ICT sup-ported pedagogical models (PhD thesis). Department of Computer Science, College ofEngineering, Ponticia Universidad Catlica de Chile.

    Rodrguez, P., Nussbaum, M., & Dombrovskaia, L. (2012). Evolutionary development: amodel for the design, implementation and evaluation of ICTs for education programs.Journal Computer Assisted Learning, 28, 8198.

    Romn, M. (2010). Desarrollo de un Estndar de Evaluacin para los Modelos de Inform-tica Educativa (MIE): Informe nal. Retrieved from http://intranet.redenlaces.cl/index.php?id=11414&no_cache=1&uid=4087

    Roschelle, J., Rafanan, K., Bhanot, R., Estrella, G., Penuel, W., Nussbaum, M., et al. (2010).Scaffolding group explanation and feedback with handheld technology: Impact on studentsmathematics learning. Educational Technology Research and Development, 58, 399419.

    Rumble, G. (2001). Analysing costs/benets for distance education programmes. The Com-monwealth of Learning. Retrieved from http://www.col.org/PublicationDocuments/pub_KS2001-02_costs.pdf

    Snchez, J., & Alarcn, P. (2003). Multimedia Project en Chile: Diseo y resultados [Multi-media project in Chile: design and results]. Presented at the Taller Internacional de Soft-ware Educativo, Santiago, Chile. Retrieved from Multimedia Project website: http://mmpchile.c5.cl/pag/publi/multimedia_project_diseno_resultados.pdf

    Scrimshaw, P. (2002). Total cost of ownership: A review of the literature. London: Depart-ment for Education and Skills.

    Slavin, R.E. (2008). Perspectives on evidence-based research in education What works?Issues in synthesizing educational program evaluations. Educational Researcher, 37,514.

    Snider, R.C. (1992). The machine in the classroom. Phi Delta Kappan, 74, 316323.Steinberg, D.C. (2004). The business model of e-learning in UK higher education. Industry

    and Higher Education, 18, 187198.

    314 P. Rodrguez et al.

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

  • Stufebeam, D.L., & Shinkeld, A.J. (2007). Evaluation theory, models, and applications.San Francisco, CA: John Wiley and Sons.

    Sunkel, G. (2006). Las Tecnologas de La Informacin y la Comunicacin (Tic) en la educa-cin en Amrica Latina. Una exploracin de indicadores [ICT in Latin America: explor-ing indicators]. CEPAL Politicas sociales (Division de Desarrollo Social, CEPAL,Naciones Unidas). United Nations.

    Sutherland, R., Armstrong, V., Barnes, S., Brawn, R., Breeze, N., Gall, M., et al. (2004).Transforming teaching and learning: Embedding ICT into everyday classroom practices.Journal of Computer Assisted Learning, 20, 413424.

    Tearle, P. (2003). ICT implementation: What makes the difference? British Journal of Edu-cational Technology, 34, 567583.

    Tearle, P. (2004). A theoretical and instrumental framework for implementing change in ICTin education. Cambridge Journal of Education, 34, 331351.

    Tondeur, J., van Braak, & Valcke, M. (2007). Curricula and the use of ICT in education:Two worlds apart? British Journal of Educational Technology, 38, 962976.

    Tondeur, J., Valcke, M., & van Braak, J. (2008). A multidimensional approach to determi-nants of computer use in primary education: Teacher and school characteristics. Journalof Computer Assisted Learning, 24, 494506.

    Toyama, K. (2010). Can technology end poverty? Boston Review (November/December).Retrieved from http://bostonreview.net/BR35.6/ndf_technology.php

    Toyama, K. (2011). There are no technology shortcuts to good education. EducationalTechnology Debate, infoDev-UNESCO. Retrieved from http://edutechdebate.org/ict-in-schools/there-are-no-technology-shortcuts-to-good-education/

    Trucano, M. (2005). Knowledge maps: ICTs in education. ICT and Education Series.Washington, DC: infoDev/World Bank. Retrieved from http://www.infodev.org/en/Publi-cation.8.html

    Tubin, D., Mioduser, D., Nachmias, R., & Forkosh-Baruch, A. (2003). Domains and levelsof pedagogical innovation in schools using ICT: Ten innovative schools in Israel. Educa-tion and Information Technologies, 8, 127145.

    Twigg, C. (2002). Improving quality and reducing costs: Designs for effective learning usingICT. Observatory on Borderless Higher Education.

    UNESCO. (2008). ICT competency standards for teachers Competency standards modules.Paris: UNESCO. Retrieved from http://cst.unesco-ci.org/sites/projects/cst/The%20Stan-dards/ICT-CST-Competency%20Standards%20Modules.pdf

    Valdivia, R., & Nussbaum, M. (2007). Face-to-face collaborative learning in computer sci-ence classes. International Journal of Engineering Education, 23, 434440.

    Valdivia, R., & Nussbaum, M. (2009). Using multiple choice questions as a pedagogic modelfor face-to-face CSCL. Computer Applications in Engineering Education, 17, 8999.

    Van den Akker, J. (2003). Curriculum perspectives: An introduction. In J. van den Akker,U. Hameyer, & W. Kuiper (Eds.), Curriculum landscapes and trends (pp. 110). Dordr-echt: Kluwer.

    Van den Akker, J., Gravemeijer, K., McKenney, S., & Nieveen, N. (2006). Educationaldesign research. London: Routledge.

    Wagner, D.A., Day, B., James, T., Kozma, R.B., Miller, J., & Unwin, T. (2005). Monitoringand evaluation of ICT in education projects: A handbook for developing countries. ICTand Education Series. Washington, DC: infoDev/World Bank. Retrieved from http://www.infodev.org/en/Publication.9.html

    Warschauer, M., Knobel, M., & Stone, L. (2004). Technology and equity in schooling:Deconstructing the digital divide. Educational Policy, 18, 562588.

    Watson, D.M. (2001). Pedagogy before technology: Re-thinking the relationship betweenICT and teaching. Education and Information Technologies, 6, 251266.

    Waxman, H., Lin, M., & Michko, G.M. (2003). Meta-analysis: Effects of educational tech-nology on student outcomes. Learning Point Associates. Retrieved from http://www.ncrel.org/tech/effects2/

    Zurita, G., & Nussbaum, M. (2004a). A constructivist mobile learning environment supportedby a wireless handheld network. Journal of Computer Assisted Learning, 20, 235243.

    Zurita, G., & Nussbaum, M. (2004b). Computer supported collaborative learning using wire-lessly interconnected handheld computers. Computers & Education, 42, 289314.

    Technology, Pedagogy and Education 315

    Dow

    nloa

    ded

    by [U

    nivers

    ity of

    Birm

    ingha

    m] at

    13:56

    03 O

    ctobe

    r 201

    3

Recommended

View more >