Download pdf - Development of an Instrument to Measure Preservice Teachers' Technology Skills, Technology Beliefs, and Technology Barriers

Jl. of Technology and Teacher Education (2013) 21(2), 179-202

Assessing Pre-service Teachers’ Information and Communication Technologies Knowledge

Cindy KovaliK, Chia-ling Kuo, and aryn KarpinsKiKent State University, USA

[email protected]@kent.edu

[email protected]

The impact of a redesigned educational technology course on preservice teachers’ knowledge and skills with regard to information and communications technology as defined by isTE’s national Educational Technology standards for Teachers (nETs-T) was investigated in this study. Two changes were made to the course: (1) a commercially avail-able technology assessment aligned with isTE’s nETs-T was administered, and (2) students completed observations of a technology-rich classroom. results indicated that the preservice teachers made significant progress in technology knowledge in all five standard areas of the nETs-T; how-ever, some performance indicators of the standards may not have been adequately addressed in the course. student reflec-tions from the observation experience provided evidence that preservice teachers found it helpful for their future teaching. although the majority of preservice teachers agreed that the observations were beneficial, a few preservice teachers had concerns about the integration of technology into lessons, es-pecially when trying to determine if technology integration is necessary, dealing with unexpected technical problems, managing the classroom, and assisting children with special needs.

pre-service teachers need to gain the knowledge, skills, and attitudes in their education programs that will prepare them to effectively integrate

180 Kovalik, Kuo, and Karpinski

information and communication technologies (iCT) in their future class-rooms. one key to ensuring pre-service teachers have these skills is to in-clude well-designed technology-rich courses throughout teacher education programs that emphasize pedagogy as well as embedding and modeling ap-propriate technology knowledge, skills, and integration (Koc & Bakir, 2010; Koh, 2011; polly & Moore, 2008).

a second key to ensuring pre-service teachers have strong iCT knowl-edge and skills is through adequate assessment strategies (stobaugh & Tas-sell, 2011). self-assessment surveys are often the instruments of choice for assessing pre-service and in-service teacher iCT knowledge and skills (at-kay, 2009; Brush, glazewski, & hew, 2008; hartshorne, Miller, & gretes, 2009; Minaidi & hlapanis, 2005); however, these types of surveys rely on self-determination of skills and knowledge rather than validated and reliable tests. Thus, few studies report on pre-service teachers’ iCT skills as mea-sured by a skills-based test.

opportunities for pre-service teachers to observe effective technol-ogy integration in actual classrooms is a third key in providing pre-service teachers with the experiences needed to help them learn how to effective-ly teach with technology. although studies have found many pre-service teachers possess strong beliefs about the value of using technology for learning and feel confident in their technology skills, many of these pre-ser-vice teachers do not feel prepared to effectively integrate technology into their future classrooms (lei, 2009; polly & Moore, 2008). This feeling of being ill-equipped to effectively use technology with students may be the result of pre-service teachers having few opportunities to observe in-service teachers using technology effectively in actual K-12 classrooms.

To ensure these three factors were part of the education program at our university, the education technology course that already contained technol-ogy-rich project-based assignments, was modified. The modifications in-cluded the addition of a commercially-available assessment instrument to measure pre-service teachers’ iCT knowledge and skills and the addition of required observations of classroom activities in a technology-rich class-room. Both the course and the assessment instrument were based on the in-ternational society for Technology in Education’s (isTE) national Educa-tional Technology standards for Teachers (nETs-T). study results indicate that students made significant progress in their iCT knowledge and skills and that they recognized the value of observing in-service teachers using multiple technologies with K-12 students.

Assessing Pre-service Teachers’ Information and Communication 181

Literature Review

The dual nature of iCT within a teacher education program makes it complex and multifaceted. For not only do pre-service teachers need to gain content knowledge about the subjects they will teach and the methods and strategies to effectively teach those subjects, but future teachers also need to master both iCT knowledge for themselves and pedagogical knowledge about how to integrate technology in K-12 classrooms (Markauskaite, 2007). This combination of knowledge and skills affords teachers abilities to work effectively in helping their students attain both curriculum and iCT learning goals.

Further, researchers have found that when teacher education courses effectively integrate technology within content areas, encourage collabora-tive efforts with in-service teachers, and include field experiences that in-clude specific technology uses, pre-service teachers are more predisposed to use technology in their future classrooms (Chen, 2010; Xiaodong, Mciner-ney, & Frechtling, 2010). Kim & hannafin (2011) addressed the perceived lack of observation opportunities by developing web-based case studies that helped pre-service teachers refine and strengthen their understanding of what it means to teach with technology. in another study, a combination of observations of technology use by university instructors and cooperating teachers and hands-on experiences with technology resulted in increased confidence in computer skills for pre-service teachers (Fleming, Motamedi, & May, 2007). Thus, high computer self-efficacy also may influence future integration of technology.

one way that computer self-efficacy may be increased is through ef-fective interactions between classroom instructors and pre-service teach-ers in an educational technology course designed to teach technology skills (Koh & Frick, 2009). in the Koh and Frick study, pre-service teachers’ com-puter self-efficacy was supported and enhanced when instructors used effec-tive strategies including show and tell, progress checking, and inviting sug-gestions from students. The researchers hypothesized that the use of these types of strategies, by positively impacting pre-service teachers’ comfort and self-efficacy with technology, may also be a positive factor leading to increased technology integration when these pre-service teachers become teachers themselves (Koh & Frick, 2009).

in a study by Teo (2009), significant relationships were found between pre-service teachers’ perception of their self-efficacy in both basic computer skills and using technology for pedagogical purposes, and how they intend-ed to use technology in their future classrooms. in other words, two factors,


namely, belief that one possesses good technology skills and that one has a good understanding of how technology can be used for pedagogical pur-poses, are “significant predictors” of how pre-service teachers intend to use technology with students (p. 13).

regardless of how much interest pre-service teachers have in intending to integrate technology in their future classrooms, that interest is tempered by reservations they may have about the integration process, their limited skills in using collaborative tools such as wikis and blogs, their lack of ad-vanced computer skills, and their limited use of technology in their own K-12 experiences (lei, 2009). To address these issues, lei suggested pre-service teachers have more exposure to the use of a variety of technologies that can be used to support teaching and learning, including subject-specific technologies and assistive technologies. pre-service teachers may also ben-efit from knowledge of the conditions necessary for technology integration (lei, 2009), and how technology, pedagogy, and content knowledge can work together for meaningful learning (Mishra & Koehler, 2006).

recognizing the need to teach and assess pre-service teachers’ basic and advanced computer skills, to relate those skills to specific content areas and K-12 curriculum standards, and to provide pre-service teachers with op-portunities to observe in-service teachers as they actively and effectively in-tegrate technology into K-12 classrooms, the required educational technolo-gy course at a large Midwestern university was modified to include a skills-based assessment and required observations in a technology-rich classroom.

Research Question

The purpose of this study was to assess the general impact of a rede-signed educational technology course. First, changes in teacher information and Communications Technology (iCT) knowledge were assessed via the administration of a survey based on the international society for Technol-ogy in Education’s (isTE) national Educational Technology standards for Teachers (nETs-T). second, the impact of a redesigned educational tech-nology course was examined through classroom observations and reflective papers. The specific research question was: (1) how does a redesigned edu-cational technology course influence pre-service teacher iCT knowledge as defined by isTE-T standards?


MeThods

Participants

undergraduate pre-service teacher education students (N = 100) en-rolled in an educational technology course at a large, Midwestern univer-sity were recruited to participate in the study. There were 66 females and 34 males. The majority of participants were juniors (n = 48), followed by sophomores (n = 27), seniors (n = 23), and freshmen (n = 2). Finally, the largest group of majors were Early Childhood Education (n = 27), with inte-grated social studies (n = 12) and Mathematics (n = 11) as the next highest categories. other majors represented included special Education, physical Education, art Education, and intervention specialist.

setting

The Educational Technology Course. Educational Technology is an introductory technology course with the aim of increasing pre-service teach-ers’ technological competency and preparing them to use information and communication technology appropriately and effectively in their future classrooms. students enrolled in the Educational Technology course major in subjects across the disciplines and have had very different prior experi-ences with computer technology. Most of the students enrolled are in their second or third year of study. These students use technology for commu-nication purposes (e.g., cell phone, text messaging), personal pleasure, and networking (e.g., ipod, Facebook). however, most of these students have never thought about applying any of these technologies in the classroom or integrating technology into teaching, or they have reservations about how and when to use technology with students. Their ideas of teaching typically derive from the way they were taught, which may not have included much technology.

To help students acquire necessary technology competencies, increase their knowledge of various technology tools, and connect technology to their proposed content areas, project-based learning is used in the course. Based on students’ proposed content areas and grade level(s), they design technology-integrated lesson plans and create teaching materials, including computer-based learning modules, videos, and WebQuests. in addition, stu-dents in the course identify web resources for their specific target learners and explore and study Web 2.0 tools that can be used in the classroom. an electronic portfolio is the culminating assignment for the course.


Materials

Simple Assessment. The measure of pre-service teacher technology knowledge used was the Teacher Technology proficiency assessment. The assessment was designed by simple K-12 (infosource, inc.) for classroom educators to ensure that teachers are effectively using technology to increase student achievement. licenses for the simple assessment instrument were purchased from simple K-12 (infosource, inc.), and, as a licensed product, we are unable to include samples of the web-based assessment. The assess-ment consists of a pretest and posttest (i.e., the same test given twice in one semester). Each test included the same, randomized 60 online, interactive, multiple-choice, performance-based questions aligned with the 2008 nETs-T. in terms of content validation, the test was developed with the help of educators in technology from all over the united states who reviewed the questions and gave their feedback. The test was revised based on the feed-back and given to the educators to review again. There were three rounds of review.

Each question was identified as addressing specific nETs-T standards and their performance indicators. The questions involve demonstrating flu-ency in multiple productivity programs, presenting an understanding of incorporating digital tools to enhance teaching and learning, choosing the best technological tools to maximize content learning in local, global, and online communities, modeling legal and ethical use of digital information and technology, and indicating possible engagement in professional growth and leadership. The test provides instant feedback and detailed reporting of results. Twelve questions addressing each of the five standards comprise the test. grant funds were used to cover the cost of the test.

Procedure

Intervention. The educational technology course was redesigned to provide for the administration of a commercially-developed iCT assessment instrument and to provide students an opportunity to observe in-service teachers actively using technology with students in a technology-rich class-room environment, namely the aT&T Classroom housed on the university campus. The redesign effort consisted of two course modifications, (1) pre- and post-administration of Teacher Technology proficiency assessment, (2) observation in the aT&T Classroom coupled with a reflective paper assign-ment.


Completing the Teacher Technology Proficiency Assessment. The pre-service teachers took the pretest during the first week of the semester (september, 2011) and the posttest at the end of the semester (december, 2011). The pre and posttests were the same, randomzied version of the Teacher Technology proficiency assessment. students were awarded credit based on the completion of the assessment instead of their actual test scores.

Classroom Observation on Technology Integration. The aT&T classroom, which opened in spring 1998 on the main university campus, is a learning and research laboratory for students and K–12 teachers to learn and teach with and through technology. an observation area is directly ad-jacent to the classroom. one-way glass partitions allow observation of the classroom without K-12 students being aware they are being observed. The observation area is equipped with adjustable recording devices and micro-phones enabling observers to listen to specific conversations and to see how individual students use the available technologies. a morning class and an afternoon class are offered each semester for six weeks at a time. The learn-ers in the aT&T classroom come from K–12 schools within a reasonable driving distance of the university.

The pre-service teachers in the educational technology courses were required to complete two 75-minute observations and write a two-page re-flective paper on their observation experience. Because of the conflict be-tween the class times of the educational technology course and the times when K-12 students were attending school in the aT&T classroom, not all educational technology students were able to observe the aT&T classroom in real time; moreover, the aT&T observation room is unable to accommo-date a class of 24 observers at one time. Thus, the aT&T classroom obser-vation took place using three different methods: (1) in-person observation using the observation room adjacent to the aT&T Classroom, (2) observa-tion through polycom technology, and (3) password protected viewing of pre-recorded aT&T Classroom sessions. Each aT&T classroom session was different in terms of the students and teachers that were in the aT&T class-room, the teaching strategies used, the types of technology being used, the learning activities in which the students were engaged, and types of the in-teractions between K-12 students and their teachers.

data Analysis

Quantitative data were analyzed with basic descriptive statistics such as means and standard deviations. Wilcoxon signed ranks Tests were con-


ducted to examine differences between pre- and post-test scores. Qualita-tive data were analyzed by identifying patterns in student responses to open-ended questions in the post-observation reflection papers (stake, 1995).

ResuLTs

Assumption Checking

one of the main assumptions of pre- and post-test analysis is that the scores on the dependent variable are normally distributed in each of the two populations. This test is reasonably robust to a violation of this assumption (lomax, 2007). For each of the analyses below (i.e., differences between the standards and performance indicators), normality was checked for each variable (i.e., the pre-test and post-test score totals for each standard and performance indicator). There were slight departures from normality when histograms and skewness and kurtosis statistics were examined, with some variables having a slight negative skew (i.e., more higher scores than lower scores). There were more slightly negatively skewed variables in the post-test than the pretest, which is to be expected due to mastery and learning of the content area (i.e., technology knowledge). Thus, to be conservative, the Wilcoxon signed ranks Test was performed (i.e., instead of t-tests). This test is recommended when the data are non-normal with extreme outliers (Wilcoxon, 1945).

Quantitative data Analysis

Teacher Technology Proficiency Assessment. There are five main na-tional Educational Technology standards for Teachers (nETs-T): (1) Facili-tate and inspire student learning and creativity, (2) design and develop dig-ital-age learning experiences and assessments, (3) Model digital-age work and learning, (4) promote and model digital citizenship and responsibil-ity, and (5) Engage in professional growth and leadership. grouping the 60 questions by these standards (i.e., 12 questions per standard, for a range of 0 to 12), pre- and post-test differences were examined (N = 100). see Table 1 for descriptive statistics of the pre- and post-test data.


Table 1descriptive statistics of pretest and posttest data from the Teacher

Technology proficiency assessment (n = 100)

Pretest Posttest

Standard M SD Min Max M SD Min Max

Total 39.56 8.55 9 55 44.77 7.65 10 56

1 7.75 1.98 3 12 8.55 1.79 2 12

1A 1.49 .77 0 3 1.72 .75 0 3

1B 2.04 .91 0 3 2.25 .69 0 3

1C 2.15 .72 0 3 2.39 .75 0 3

1D 2.10 .66 0 3 2.19 .68 0 3

2 7.62 1.99 1 11 8.78 1.83 2 12

2A 1.90 .76 0 3 2.10 .60 0 3

2B 2.48 .75 0 3 2.70 .64 0 3

2C 1.99 .81 0 3 2.26 .75 0 3

2D 1.26 .80 0 3 1.72 .68 0 3

3 7.52 2.10 0 11 8.36 1.93 0 11

3A 2.37 .80 0 3 2.64 .69 0 3

3B 1.44 .78 0 3 1.67 .77 0 3

3C 2.00 .76 0 3 2.25 .73 0 3

3D 1.72 .57 0 3 1.81 .66 0 3

4 9.01 2.05 3 12 10.29 1.94 1 12

4A 2.71 .50 1 3 2.86 .40 1 3

4B 1.87 .88 0 3 2.33 .83 0 3

4C 2.49 .69 0 3 2.61 .68 0 3

4D 1.94 .90 0 3 2.49 .67 0 3

5 7.81 2.65 0 12 8.79 2.00 3 12

5A 2.29 .90 0 3 2.51 .67 0 3

5B 2.27 .84 0 3 2.41 .77 0 3

5C 1.75 1.05 0 3 2.01 .87 0 3

5D 1.61 .82 0 3 1.86 .73 1 3


Wilcoxon signed ranks Tests were conducted to determine if there were statistically significant differences between the pre-test and post-test means for each standard. The Bonferroni correction was applied (i.e., α = .05/5 = .01). all tests indicated that the pre-test and post-test means were significantly different across all standards (p < .001) indicating that pre-ser-vice teachers gained a significant amount of technology knowledge in each standard area on the test of technology proficiency (see Table 2).

Table 2Wilcoxon signed ranks Tests: pretest and posttest of Teacher Technology

proficiency assessment (n = 100)Wilcoxon Signed Ranks Tests

Standard Z df p d

Total -6.67 99 .000***

1 -3.65 98 .000*** -.38

1A -2.44 99 .015

1B -2.13 99 .033

1C -2.81 97 .005** -.30

1D -1.21 99 .225

2 -5.24 99 .000*** -.63

2A -2.34 98 .019

2B -2.79 99 .005** -.30

2C -3.27 99 .001** -.34

2D -4.31 99 .000*** -.48

3 -4.11 99 .000*** -.43

3A -2.73 98 .006** -.30

3B -2.33 99 .020

3C -2.73 97 .006** -.29

3D -1.41 97 .158

4 -5.66 98 .000*** -.68

4A -2.47 98 .014

4B -4.13 98 .000*** -.46

4C -1.49 98 .135

4D -5.75 98 .000*** -.72

5 -3.98 97 .000*** -.42

5A -2.70 97 .007** -.28

5B -1.29 97 .197

5C -2.24 95 .025

5D -2.27 97 .024

Note. ***p < .001; **p < .01. The Bonferroni correction was applied for each standard (i.e., α = .05/5 = .01) and for each performance indicator (i.e., α = .05/4 = .013). For Cohen’s d (i.e., Effect size), ≈ .30 is a small effect, ≈ .50 is a medium effect, and ≈.70 is a large effect.


For each of the five nETs-T standards, there are four performance indicators (i.e., a, B, C, d; see appendix). grouping the 60 questions by these performance indicators (i.e., three questions per performance indica-tor, for a range of 0 to 3), pre- and post-test differences were examined (N = 100). Wilcoxon signed ranks Tests were again conducted to determine if there were statistically significant differences between the pre-test and post-test means for each performance indicator. The pre-test and post-test means were significantly different for performance indicators 1C (p = .004), but not for 1a, 1B, and 1d (p > .013). For standard 2, the pre-test and post-test means were significantly different for performance indicators 2B, 2C, and 2d (p < .01), but not for 2a (p > .013). For standard 3, the pre-test and post-test means were significantly different for performance indicators 3a and 3C (p < .01), but not for 3B and 3d (p > .013). For standard 4, the pre-test and post-test means were significantly different for performance indica-tors 4B and 4d (p < .013), but not for 4a and 4C (p > .013). For standard 5, the pre-test and post-test means were significantly different for performance indicators 5a (p < .01), but not for 5B, 5C, and 5d (p > .013; see Table 2).

Qualitative data Analysis

Four main themes emerged from two-page reflection papers that stu-dents in the educational technology course wrote. The four themes were technology integration, unanticipated observations, suggestions for im-provement, and benefits of observing a technology-rich classroom.

Technology Integration. a majority of the pre-service teachers were positive about observations of teachers incorporating different types of technology in their lessons and K-12 students reacting to the use of tech-nology. pre-service teachers indicated that technology helped to capture stu-dents’ attention and helped them learn. as one student commented, “stu-dents seem to enjoy working on assignments when they are able to use these [technological] devices because it captures their attention and helps them learn.” The pre-service teachers also noticed that the K-12 students worked together and relied less on the help of their teacher. one student wrote, “The students often turned to each other when they had questions before ask-ing the teacher.” several pre-service teachers noted that the K-12 students seemed more independent in their learning. one student stated, “The use of technology in the classroom gives students more of a sense of indepen-dence… i feel that it also gives the students the chance to explore things on their own and for themselves.” observing how in-service teachers integrat-


ed technology into the lessons also introduced new ideas to the pre-service teachers. representative statements included,

By observing what really interests the students, i can discover more ways in which learning with technology will be motivational. Behind all the fun and games, a new level of learning is reached including deeper understanding, and critical thinking.

and “By observing these classrooms it really helped me to see all the differ-ent ways that i personally could integrate into my own classroom one day.”

Unanticipated Observations. The K-12 students’ familiarity and con-fidence while using technology were pointed out by most of the pre-service teachers. The ways that the K-12 children responded to the technology dif-fered completely from what the pre-service teachers had expected. The pre-service teachers were also surprised by the concentration and involvement that the children exhibited. one pre-service teacher stated,

i cannot believe how much the kids know about technology. This makes me realize how much i need to learn about technology be-fore i become a teacher so i can be on my students’ level or at least closer to it.

The pre-service teachers were surprised “…how quickly [the K-12 stu-dents] understood how to use…[programs]” and that they tended to stay on task. as one student wrote, “i would like to know how the teachers get the students to stay on task for so long.” another student captured the sentiment of many students by writing, “i was overly shocked at the amount of moti-vation, desire, and fun each student had in participating in the technological activities.”

observing K-12 students in a classroom setting provided an opportu-nity for pre-service teachers to see students actively engaged in using tech-nology to complete projects. as one student noted,

i was very surprised at how well the students performed on the computer. Most of them knew what they were doing and were able to finish their project independently. one student explained how to record their [sic] voice and listen to the recording once she was finished. i was impressed that the young children were able to un-derstand how to perform the in-depth tasks that they were assigned.

Being able to observe technology integration without physically being pres-ent in a classroom also provided insight into classroom management issues and helped pre-service teachers reflect on how they might address these is-sues when they become teachers.

Suggestions for Improvement. regarding what might have been done differently in the observed lessons, study participants commented mainly


on better classroom discipline and better ways to answer students’ techni-cal questions. in some sessions that the pre-service teachers observed, they noticed that the children sometimes were off task or that they chatted with their friends on unrelated topics. one pre-service teacher commented, “i would not leave the children for more than just a few minutes. When the teacher did this, the kids got a little bit off task and wandered around the classroom.” some pre-service teachers observed that one teacher had diffi-culty responding to all the children’s questions in a timely manner and of-fered solutions: “When the students started the project, i would have given them a step-by-step handout with screen shots of the online tool they were using so there wouldn’t be so many questions.” a second solution was “i would have gotten the entire class’s attention so the questions could be asked in front of everyone. hopefully, this would eliminate having to repeat the same answer to several different students”

Benefits of Observation. overall, the pre-service teachers thought the observation experience was helpful and beneficial to their future teaching. as one pre-service teacher stated,

overall i feel like this really was a great experience and taught me a lot about implementing technology into a classroom. When i think about teaching [in] an elementary school classroom, i think about my experience, which was quite a few years ago. i never knew that there really were this many options that students could do in class.

The pre-service teachers felt the experience was both rewarding and inter-esting because they were able to see an in-service teacher engaging students in a technology-enhanced lesson without the K-12 children being aware that they were being observed. The majority of the pre-service teachers ex-pressed that the observations opened their eyes to how many different ways technology can be used and inspired them to use technology in the class-room. a representative statement was, “These observations got me thinking about how i want to run my future classroom and how technology should be integrated into it.” another pre-service teacher wrote, “all in all this was a good experience and gave me a good look at not only technology but teach-ing in general.”

The observation experience also helped the pre-service teachers be-come more aware of potential problems when using technology. as one pre-service teacher mentioned, “This [observation] also allowed me to be aware of potential off task behavior and how to minimize it when using the tech-nology.”

Classroom management also was apparent in more negative opinions related to the use of technology in the classroom. some of the pre-service


teachers argued that too much emphasis was placed on technology and they voiced the belief that traditional instructional methods are equally as effec-tive. Moreover, technical problems, students’ lack of discipline while the technology was being fixed, and lack of assistance for children with special needs were concerns that these pre-service teachers raised. in one instance, a pre-service teacher noted how technical problems impacted the lesson by writing, “The lesson could have been rather productive, but the use of tech-nology hindered that. More time was focused on how to fix the computers or keeping everyone on track, than was used to actually learn the lesson.” some pre-service teachers questioned the use of technology with com-ments including, “The teacher used the sMarTBoard but would probably have been better off just drawing on a white board,” “i feel that technology can be used in a helpful manner, but that it is being forced into a context in which it is not needed,” and “…because there is a new way to do things does not always mean it is the best way to do it.”

pre-service teachers were able to observe issues related to classroom management and K-12 student lack of self-discipline that caused one stu-dent to write,

i noticed that in the second video i watched, the teacher had a more difficult time controlling the students. There was a lot of chatter and yelling. Many students were out of their seats, and the teacher seemed overwhelmed. i believe that in a classroom like this, not all teachers can succeed because most are trained to work in a traditional classroom.

Concern also was expressed about providing equal access for all stu-dents. as one student commented,

When everyone knows what they are doing in the class and kids go and start working in their assignment, the kids with learning disabilities are the ones usually left behind. integrating technology might be hard with these kids because they need a teacher there to explain the whole assignment to them.

discussion

The statistical results of the Teacher Technology proficiency assess-ment provide empirical evidence that the pre-service teachers in this study gained a significant amount of technology knowledge and skills in each of the five general isTE nETs-T standards. The project-based assignments in the education technology course afforded pre-service teachers the oppor-


tunity to increase their technology skills in multiple applications while en-couraging them to focus on K-12 curricular goals when designing technol-ogy-integrated lessons. The classroom observations provided a real-world context for students to see and better understand technology integration. as one student wrote,

after watching the aT&T classroom videos, i finally found out how to incorporate a lot of technology into a classroom. Before watching these videos, i could never imagine how to do this. now i have ideas on how to use technology in a classroom when i be-come a teacher.

another student commented, “The classroom observation was a great way to see how the teacher interacted with the students, how she presented her material, and how the students interacted with one another as well as [with] their new environment.”

although pre-service teachers made significant progress in iCT knowl-edge and skills, some isTE performance indicators were identified as pos-sible areas for improvement. For standard 1, pre-service teachers may need more guidance and experience in ways to support creativity and innovative thinking (1a), to use real-world issues in lessons (2B), and to use collab-orative tools to build knowledge (1d). Because pre-service teachers in the educational technology course were still relatively early in their education coursework it is not surprising that they struggled with knowing how to incorporate creativity and real-world issues into lessons, or how to model collaborative knowledge construction. pre-service teachers may need more guidance in knowing how to find and identify real-world issues that can be used as the basis for lessons. Encouraging the use of online resources such as the union of international associations’ database of world problems (http://www.uia.be/sites/uia.be/db/db/x.php?dbcode=pr&go=) may result in greater awareness of real-world issues that subsequently can be used to cre-ate lesson plans or instructional materials.

in the area of collaborative knowledge construction, pre-service teach-ers may be adept at using social media, but they typically have had little ex-perience with using these types of tools for learning (lei, 2009). Expanding the use of blogs and wikis in the educational technology course or encour-aging participation in online learning communities with in-service teachers may provide the types of experiences pre-service teachers need in order to learn how to model the learning process using technology tools.

prior to taking the education technology course, some of the pre-service teachers in the study may have given little thought to using technology in the classroom, even though they were aware that they need to have good


technology skills themselves. as one student stated, “i was not planning on using much technology, but now i know that using technology helps engage students to learn better.” To address possible weakness in the 2a nETs-T performance indicator, an investigation of strategies that can help to iden-tify K-12 curricular areas where the effective integration of technology may promote student learning could be added to the course through online or classroom discussion. another option would be to design and develop an as-signment where pre-service teachers adapt an existing learning experience to include appropriate digital tools or resources.

performance indicator 3d, Model and facilitate effective use of current and emerging digital tools to locate, analyze, evaluate, and use information resources to support research and learning, is related to information lit-eracy, an area of weakness for many college students (allen, 2007; Caspers & Bernhisel, 2007; Kolowich, 2011). The information literacy skills of the pre-service teachers in this study may align with research findings that have indicated students may be able to find information, but may not possess ad-equate skills in knowing if they have found quality and relevant resources, or in knowing how to critically review that information in order to use it appropriately and effectively (Knight, 2005; Morrison, 1997; Williams & Wavell, 2007). increasing the exposure of pre-service teachers to informa-tion literacy skills, and soliciting help from university librarians for sugges-tions on ways to incorporate information literacy into existing assignments, such as the WebQuest, may positively affect the information literacy skills of pre-service teachers.

digital citizenship is represented in performance indicator 4C, an area that did not show statistically significant differences. although pre-service teachers are well-versed in using electronic media for communication and social interactions, heightened awareness of digital etiquette and responsi-ble use of social media may be achieved in the education technology course through course guidelines related to online communication. These guide-lines could contain information on appropriate electronic exchange of infor-mation, protection and security of personal information, respect for oneself as well as others in online environments, and adherence to copyright laws related to intellectual property.

Examining the remaining performance indicators that did not show sta-tistical improvement (3B, 5B, 5C, and 5d) revealed that they were primarily geared toward classroom activities and interactions expected to occur in a classroom or at school. For the pre-service teachers to meet these indicators without actually teaching or being in a school environment was challenging. Thus, it is understandable why the pre-service teachers failed to make sig-


nificant gains in these areas. Through class discussions, the course instructor could explain expected behaviors and activities to the pre-service teachers, identify the performance indicators that showed no statistical improvement, and provide guidance on meeting them in the future.

reflective statements from pre-service teachers’ observation papers on the classroom activities in a technology-rich classroom implied that the pre-service teachers needed this kind of experience. not only did the ob-servation experience model teaching and learning with technology but it also changed the pre-service teachers’ opinions about using technology in the classroom. one student majoring in mathematics wrote in her electronic portfolio:

When i first signed up for this course, i thought there would be no real use [of technology] in mathematics because it seems like such a “paper and pencil” subject. But i learned very quickly that there are plenty of neat things you can use technology for in math and i think that the technology would definitely help kids learn and stay engaged in the subject. Being able to see some of the technology at work in the aT&T classroom was also helpful because i could actually picture myself using these things in an effective way. i’m very excited to try all kinds of technology in my future classroom.

Through the observation experience the pre-service teachers realized that to-day’s children are naturally technology-savvy (“i have a new outlook of the upcoming generation. They are extremely intelligent in technology, which means us [sic] as pre-service teachers need to catch up to them and give them the opportunity to live up to their potential”). These pre-service teach-ers learned that technology captures students’ attention and engages stu-dents in learning, and that integrating technology into lessons may occur in numerous ways. as one student stated, “The entire aT&T distance learning experience was very helpful. it provided insight into how professionals op-erate in the classroom setting, as well as helping me formulate some ideas of how i would like to organize my classroom in the future and incorporate technology into my lessons.”

a few pre-service teachers maintained more conservative attitudes on the integration of technology into lessons. Their concerns identified a need for teacher education programs to be mindful to address classroom manage-ment issues related to the use of technology, including ways to deal with children (a) who did not stay on task while the teacher tried to fix technical problems, (b) whose questions could not be responded in a timely matter, (c) who chatted with peers on unrelated topics, and (d) who may need ad-ditional assistance. The pre-service teachers felt that some of their obser-


vations indicated the classroom teacher at times seemed unable to control these types of situations. Teacher preparation training typically emphasizes classroom management strategies in a traditional classroom setting and may not address useful strategies in situations where technology is being used. in order to enhance pre-service teachers’ knowledge and skills with regard to technology integration, classroom and discipline management, assis-tance for inclusion students, and proper ways to handle unexpected techni-cal problems should be addressed in teacher preparation training. address-ing these classroom management issues may help pre-service teachers better plan for technology-infused lessons and activities by equipping them with strategies they can use to plan for, guard against, or prevent possible tech-nology-related problems in their future classrooms.

Based on the results of the technology assessment and the reflection pa-pers written on the observation experience, the redesigned educational tech-nology course helped pre-service teachers in this study gain iCT knowledge and skills and better prepared them to integrate technology into their future classrooms. Through reflective statements in their electronic portfolios, pre-service teachers described the impact of the redesigned educational technol-ogy course. one student wrote,

in this course, i have learned many methods of using and integrat-ing technology into my classroom. i have learned how to more ef-ficiently use powerpoint, prezi, Microsoft Word, Microsoft Excel, audacity, Windows Movie Maker, and many more useful software programs that i will no doubt incorporate into my teachings.

another student described a higher level of confidence in using technolo-gy by stating, “i feel more confident in implementing these new technolo-gies into my classroom because i know about how they work and how easy they are to use, not only for myself but also for students to use.” pre-service teachers were also aware that their observations provided opportunities to see both negative and positive aspects of technology use, as shown in this statement, “after having the opportunity to watch elementary classes utilize the aT&T classroom i could see the advantages and disadvantages and the many diverse ways to integrate technology into lessons.”

pre-service teachers in this study were also able to observe K-12 stu-dent engagement in a technology-rich environment. one student described student engagement with technology by writing,

The use of technology not only captures the students’ interests and gets them involved, but most importantly it makes learning fun. it is when the student does not realize they are learning because it is so fun and engaging that you know they are really learning and enhancing their abilities. This class really made me realize this,


and allowed me to begin to explore the multiple ways to integrate technology into teaching.

Implications

The use of a commercially-available performance-based technology skills assessment provided an objective measurement of pre-service teacher technology knowledge and skills. instead of relying on student self-assess-ment of their technology skills, the Teacher Technology proficiency assess-ment provided empirical evidence of growth in technology skills through the pre- and post-administration of the assessment and provided strong sup-port that the educational technology course was successful in helping pre-service teachers achieve learning outcomes. Colleges of education may want to investigate the implementation and use of empirical performance-based assessments as an integral part of their teacher education programs.

The addition of required observations in a technology-rich classroom environment addressed a gap identified by polly and Moore (2008), namely, that pre-service teachers typically do not have adequate exposure to K-12 classrooms where technology is being integrated in effective and efficient ways and thus “lack a sense of what technology integration looks like” (p. 26). The aT&T Classroom is a relatively unique environment due to the wide range of available technologies, the ability to observe the classroom in unobtrusive ways, and the amount of support provided to teachers who bring their students to the classroom for six weeks at a time. pre-service teachers are virtually assured of being able to see “what technology integra-tion looks like” in this type of environment, whereas if pre-service teach-ers were to travel to a K-12 school to observe technology integration, they may or may not be successful in doing so due to school schedules, access to technology, and in-service teachers’ comfort and experience with technol-ogy. Teacher education programs may want to identify strategies that will ensure all pre-service teachers are able to observe the effective use of tech-nology for learning to encourage the development of a shared and cohesive vision of the role and use of technology in education.

Limitations and Future directions

Three limitations of the study are noted. First, the Teacher Technology proficiency assessment included simulated, performance-based questions; however, the simulations did not always function properly for a few of the


students. Therefore, these students were forced to select an option that al-lowed them to continue with the assessment, regardless of whether their se-lection was, in their opinion, a correct response. in addition, some students’ test scores did not record to the system successfully. Because of these two technical issues, some test scores had to be removed, lowering the sample size of this study. second, the observation experiences did not align with all pre-service teachers’ proposed teaching subjects and grade levels. The face-to-face observation sessions and the recorded class sessions that were available online were of third and fourth grades. The pre-service teachers enrolled in the educational technology class were from multiple disciplines and had different proposed teaching grade levels. The observation experi-ence, therefore, may not have been optimum for all enrolled students be-cause some of these pre-service teachers may have benefitted more if they had been able to observe K-12 students representative of the students they plan to teach. This issue may have affected the observation experience and the expected learning outcome for this course activity.

The third limitation of this study is related to the generalizability of re-sults. Because participants in the study were a convenience sample from one university, findings are representative only of this sample and may not be generalizable to other pre-service teachers and other teacher education pro-grams.

The redesigned educational technology course helped pre-service teachers gain a significant amount of iCT knowledge and skills as defined by isTE’s nETs-T. Through observing technology integration in an actual classroom, the pre-service teachers felt more prepared to integrate technolo-gy into their future classrooms. Future studies should focus on the nETs-T performance indicators that were not met as well as on identifying strategies that may help pre-service teachers meet them. suggestions include building learning communities with in-service teachers and university faculty to ex-change information, ideas, and methods in teaching and technology integra-tion; exchanging technology integrated lesson plans; and discussing issues and concerns related to technology integration, including classroom man-agement, inclusion, and discipline policy.

Acknowledgements

The work described in this article was supported by a 2011-2012 Teacher planning grant from eTech ohio titled The ICT2 Project: Increas-ing collaboration among Teachers for Information Communication Tech-nologies.


References

allen, s. M. (2007). information literacy, iCT, high school, and college expecta-tions. Knowledge Quest, 35(5), 18-24.

atkay, s. (2009). The isTE national Educational Technology standards and prospective primary school teachers in Turkey. The International Journal of Learning, 16(9), 127-138.

Brush, T., glazewski, K. d., & hew, K. F. (2008). development of an instru-ment to measure preservice teachers’ technology skills, technology be-liefs, and technology barriers. Computers in the Schools, 25(1-2), 112-125. doi:10.1080/07380560802157972

Caspers, J., & Bernhisel, s. M. (2007). What do freshmen really know about research? assess before you teach. Research Strategies, 20, 458-468. doi:10.1016/j.resstr.2006.12.016

Chen, r. (2010). investigating models for preservice teachers’ use of technology to support student-centered learning. Computers & Education, 55(1), 32-42. doi:10.1016/j.compedu.2009.11.015

Fleming, l., Motamedi, v., & May, l. (2007). predicting preservice teacher competence in computer technology: Modeling and application in training environments. Journal of Technology and Teacher Education, 15(2), 207-231.

hartshorne, r., Miller, g., & gretes, J. (2009). Examining student perceptions of technology skills before and after an introductory educational technol-ogy course: a three-year case study. International Journal of Technology in Teaching and Learning, 5(1), 37-48.

Kim, h., & hannafin, M. J. (2011). developing situated knowledge about teach-ing with technology via Web-enhanced case-based activity. Computers & Education, 57, 1378-1388. doi:10.1016/j.compedu.2011.01.008

Knight, l. a. (2005). using rubrics to assess information literacy. Reference Ser-vices Review, 34(1), 43-55. doi:10.1108/00907320610640752

Koc, M., & Bakir, n. (2010). a needs assessment survey to investigate preser-vice teachers’ knowledge, experiences and perceptions about preparation to using educational technologies. Turkish Online Journal of Educational Technology, 9(12), 13-22.

Koh, J. h. l. (2011). Computer skills instruction for preservice teachers: a com-parison of three instructional approaches. Computers in Human Behavior, 27, 2392-2400. doi:10.1016/j.chb.2011.08.002

Koh, J. h. l., & Frisk, T. W. (2009). instructor and student classroom interac-tions during technology skills instruction for facilitating preservice teach-ers’ computer self-efficacy. Journal of Educational Computing Research, 40(2), 211-228.

Kolowich, s. (2011). study: College students rarely use librarians’ expertise. AASL Hotlinks, 10(6), 1.

lei, J. (2009). digital natives as preservice teachers: what technology prepara-tion is needed? Journal of Computing in Teacher Education, 25(3), 87-97.


lomax, r.g. (2007). An introduction to statistical concepts (2nd ed.). Mahwah, nJ: lawrence Erlbaum associates.

Markauskaite, l. (2007). Exploring the structure of trainee teachers’ iCT litera-cy: The main components of, and relationships between, general cognitive and technical capabilities. Educational Technology Research & Develop-ment, 55, 547-572. doi:10.1007/s11423-007-9043-8

Minaidi, a., & hlapanis, g. h. (2005). pedagogical obstacles in teacher training in information and communication technology. Technology, Pedagogy, and Education, 14(2), 241-254. doi:10.1080/14759390500200204

Mishra, p., & Koehler, M. J. (2006). Technological pedagogical content knowl-edge: a framework for teacher knowledge. Teachers College Record, 108(6), 1017-1054. doi:10.1111/j.1467-9620.2006.00684.x

Morrison, h. (1997). information literacy skills: an exploratory focus group study of student perceptions. Research Strategies, 15(1), 4-17. doi:10.1016/s0734-3310(97)90003-3

polly, d., & Moore, J. (2008). The great divide: preparing preservice teachers to integrate technology effectively. in Educational Media & Technology year-book (pp. 17-30). greenwood publishing group, inc.

stake, r. E. (1995). The art of case study research. Thousand oaks, Ca: sage.stobaugh, r. r., & Tassell, J. l. (2011). analyzing the degree of technology use

occurring in preservice teacher education. Educational Assessment, Evalua-tion and Accountability, 23, 143-157. doi:10.1007/s11092-011-9118-2

Teo, T. (2009). Examining the relationship between student teachers’ self-effica-cy beliefs and their intended uses of technology for teaching: a structural equation modeling approach. The Turkish Online Journal of Educational Technology, 8(4), 7-16.

Wilcoxon, F. (1945). individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80-83.

Williams, d. a., & Wavell, C. (2007). secondary school teachers’ conceptions of student information literacy. Journal of Librarianship and Information Sci-ence, 39(4), 199-212. doi:10.1177/0961000607083211

Xiaodong, Z., Mcinerney, J., & Frechtling, J. (2010). learning after you know it all: When sTEM faculty teach teachers, who learns? Change 42(3), 24-29. doi:10.1080/00091381003730185


APPendIx

national educational Technology standards for Teachers (neTs-T)

sTandard 1: FaCiliTaTE and inspirE sTudEnT lEarning and CrEaTiviTy a. promote, support, and model creative and innovative thinking and inven-tiveness.B. Engage students in exploring real-world issues and solving authentic problems using digital tools and resources.C. promote student reflection using collaborative tools to reveal and clarify students’ conceptual understanding and thinking, planning, and creative processes.d. Model collaborative knowledge construction by engaging in learning with students, colleagues, and others in face-to-face and virtual environ-ments. sTandard 2: dEsign and dEvElop digiTal-agE lEarning EXpEriEnCEs and assEssMEnTs a. design or adapt relevant learning experiences that incorporate digital tools and resources to promote student learning and creativity.B. develop technology-enriched learning environments that enable all students to pursue their individual curiosities and become active participants in setting their own educational goals, managing their own learning, and assessing their own progress.C. Customize and personalize learning activities to address students’ diverse learning styles, working strategies, and abilities using digital tools and resources.d. provide students with multiple and varied formative and summative as-sessments aligned with content and technology standards and use resulting data to inform learning and teaching. sTandard 3: ModEl digiTal-agE WorK and lEarning a. demonstrate fluency in technology systems and the transfer of current knowledge to new technologies and situations.B. Collaborate with students, peers, parents, and community members using digital tools and resources to support student success and innovation.


C. Communicate relevant information and ideas effectively to students, parents, and peers using a variety of digital-age media and formats.d. Model and facilitate effective use of current and emerging digital tools to locate, analyze, evaluate, and use information resources to support research and learning. sTandard 4: proMoTE and ModEl digiTal CiTiZEnship and rEsponsiBiliTy a. advocate, model, and teach safe, legal, and ethical use of digital infor-mation and technology, including respect for copyright, intellectual prop-erty, and the appropriate documentation of sources.B. address the diverse needs of all learners by using learner-centered strate-gies providing equitable access to appropriate digital tools and resources.C. promote and model digital etiquette and responsible social interactions related to the use of technology and information.d. develop and model cultural understanding and global awareness by engaging with colleagues and students of other cultures using digital-age communication and collaboration tools. sTandard 5: EngagE in proFEssional groWTh and lEad-Ership a. participate in local and global learning communities to explore creative applications of technology to improve student learning.B. Exhibit leadership by demonstrating a vision of technology infusion, par-ticipating in shared decision making and community building, and develop-ing the leadership and technology skills of others.C. Evaluate and reflect on current research and professional practice on a regular basis to make effective use of existing and emerging digital tools and resources in support of student learning.d. Contribute to the effectiveness, vitality, and self-renewal of the teaching profession and of their school and community.

national Educational Technology standards for Teachers, second Edition ©2008, isTE® (international society for Technology in Education), www.iste.org. all rights reserved. used with permission.