Evaluating Technology’s Impact on Teaching and Learning

Educational Technology Integration

Contact Information

jsun@sun-associates.com 978-453-3070

www.sun-associates.com/eval/sample This presentation is linked to that page

Where Do We Stand?

Hot topics Your District State National

Your expectations for today

Workshop Goals

To overview a scheme for creating a formative evaluation of technology’s impact

To review related issues and developments visa vis our experience in a particular school district (Fayette County, KY)

To create an action plan for conducting a similar assessment in your own district

A Framework for Review

From Designing ProfessionalDevelopment for Teachers of Scienceand Mathematics, Loucks-Horsley,Hewson, Love, and Stiles. CorwinPress Inc. 1998

Why Evaluate?

To realize your investment in technology What sort of “difference” has all of this

technology made?

To continue your school or district technology planning effort Evaluation is a key component of ongoing

technology planning!

Evaluation Starts with Technology Goals

Evaluation should be rooted in a strong educational technology plan which... Is more than an infrastructure plan Focuses on technology’s impact on teachers

and students Has clear goals and objectives for what you

want to see happen

Skeletal Plan Format

Acitivity 1 Activity 2

Goal 1

Activity 1

Goal 2

Curricular Goals

Activity 1

Goal 1

Activity 1 Activity 2

Goal 2

Staff Development Goals

Activity 1

Goal 1

Activity 1

Goal 2

Community Goals

VISION

Your Goals?

Using the Plan Mapping Worksheet, map your plan in terms of vision, goals, and actions.

Goals Lead to Questions

What do you want to see happen? These are your goals

Achieving these goals requires a process Are you performing the process steps?

All of this can be measured through a formative evaluation

The Evaluation Process

Create meaningful evaluation questionsDesign relevant indicatorsOrganize those indicators in a performance

rubric which can be used to measure progress and achievement

Collect data to score the rubricsCreate a report which applies the

evaluation in a formative manner

Take a break...

FCPS Example Questions

How have students been impacted by technology integration?

Are teachers using technology in ways that match district goals for technology and use the potential for instructional technology?

Have we adequately allocated district technology resources?

Processes of Technology Evaluation

What are you looking for? Ways to make the implementation of

technology better Teaching and learning impacts — this is not

about counting machines!

Developing Indicators

What is it that you want to measure? Impact on teaching and learning Change Progress towards desired outcomes

You need indicators for progress to be measured

Indicators should reflect your school’s unique goals and aspirations Arise from your plan Rooted in your vision and goals Indicators must be indicative of your unique

environment...what constitutes success for you might not for someone else.

Indicators of Engaged Learning

Engaged learners are Responsible for their own learning Strategic problem-solvers Energized by learning Collaborative learners

These are all observable characteristicsThese indicators can help inform your

student impact rubric

Levels of Teacher Appropriation

Stage Teacher BehaviorEntry Teachers learn the "basics" of using new

technology.Adoption Teachers use new technology to support

traditional instructional methods.Adaptation Teachers integrate new technology into

traditional classroom practice, focusing onincreased student productivity andengagement through the use of tools such asword processors, spreadsheets, and graphicstools.

Appropriation Teachers focus on cooperative, project-based,and interdisciplinary work which incorporatestechnology as needed and as one of manytools.

Invention Teachers discover new uses for technologytools often by designing projects that combinemultiple technologies.

Adapted from "A Report on 10 Years of ACOT Research" (Apple Computer, 1996)

Point is... There are things we can look for in terms of student and teacher behavior.

The goal in creating indicators is to link aspects of behavior to levels of achievement

What’s Realistic?

When developing questions and indicators, what makes sense visa vis the research on technology’s potential impacts?

www.sun-associates.com/eval/resources.html

What the Research Says

CBI generates measurable increases in mastery of core content Kulik’s meta-analysis Fletcher’s evaluation of military training

Pace, repetition, and standardization promote memorization of “facts”

Kulik’s work shows that CBI did virtually nothing in terms of changing student “attitude” toward subject matter

Productivity tools -- particularly tools for writing -- encourage students to produce more and pay more attention to the quality of their work. This more free-form use of technology cuts

directly to the heart of the connection between attitude toward learning and student achievement

Students will do more of what they like to do

Technology improves the learning environment Motivates students Makes learning more exciting Makes learning more relevant to real-world

tasks

None of these benefits come solely through technology!

It gets more gray when considering “cognitive uses” Cooperative and collaborative tasks Communication Problem solving Using technology as a key element of a rich,

supportive, educational environment

Student Performance in Rich Educational Environments

Higher rates of attendance and graduation (ETS report)

Higher qualification rates for honors programsGreater development of critical thinking and

problem-solving skillsGenerally higher scores on traditional

assessmentsLower numbers of disciplinary incidents

These benefits have been observed/documented specifically in those classrooms where technology has been a key intervention...but never the only intervention!

What Other Interventions?

More teacher training Usually in relation to using the technology tools

as part of an overall reformed instructional environment

More individualized, student-centered, learning Usually as a result of problem-based,

cooperative learning environments where technology is the catalyst for change

So, What Changes?

The student outcomes from participation in a technology-enriched environment are indeed positive

Measures are usually in terms of new things that can be done versus increases on traditional assessments

Nevertheless, traditional assessments do show improvement...it just occurs over time and is part of a broader, system-wide, picture of change.

Try a Sample Indicator

Using the Developing Indicators worksheet, let’s develop a few sample indicators Work in several groups Select someone to report out Try to develop some consensus on indicators

which would work across districts

FCPS’s Indicators

Use of technology positively impacts and fosters the student’s motivation to engage in learning practices that lead to new ways of thinking, understanding, constructing knowledge, communicating results, and acquiring basic skills

The faculty and staff are proficient, knowledgeable, and current with available technology and translate that knowledge into relevant learning opportunities for students

Teachers create learning opportunities and physical environments that allow students to assume more independent roles in their own learning through their use of technology.

All teachers and learners throughout the district have sufficient access to technology-based productivity tools, on-line services, media-based instructional materials, primary sources of data, and adequate support for using these resources so as to enrich and extend their learning goals.

Rubrics

Student AchievementQuestion How has technology positively impacted student achievement?

BasicIndicator

Use of technology positively impacts and fosters the student’s motivation to engage in learning practices thatlead to new ways of thinking, understanding, constructing knowledge, communicating results, and acquiringbasic skills

Level 4 Technology is used routinely and seamlessly as a tool for learning. Technology positively impacts and fostersthe student’s motivation to engage in learning practices that lead to new ways of thinking, understanding,constructing knowledge, communicating results, and acquiring basic skills. Technology is used to inspirecritical thinking and the solving of problems relevant to real-life skills with recognition of the tradeoffs inherent inthe application of technology in society. Student work parallels the way in which professionals in the work forceuse technology.

Students exercise a high degree of personal judgement in the choice and application of technology to theirlearning. Teachers support this choice by designing and facilitating a student-centered learning environmentwhich makes use of a wide variety to technology tools.

Level 4Evidence

Technology is used in all of the ways documented in the previous levels.

In addition, the choice of technology tools used in learning is primarily student directed. Students makeappropriate choices with regard to when and where to use technology. For example…

1. The student exercises personal judgement in the maximum and most appropriate use of technology toassist problem solving, reasoning, and thinking.

2. Students demonstrate critical thinking and media literacy skills in the use of technology as a research tool.

www.fayette.k12.ky.us/central/edtech/suneval.asp

Level 3 Students regularly use technology within their learning activities. Fluent use of technology positivelyimpacts and fosters the student’s motivation to engage in learning practices that lead to new waysof thinking, understanding, constructing knowledge, communicating results, and acquiring basicskills.

Use of technology is equally student directed as it is teacher directed. Students exercise somedegree of personal choice and judgement in their use and application of technology to learning.

Level 3Evidence

As for Level 2, but considerably more advanced uses of technology are demonstrated. Forexample...

1. Students use technology tools to solve problems which require the organization and analysis ofdata (i.e., a graphing calculator, spreadsheet, database, etc.)

2. Students use software to create presentations to communicate effectively (i.e. PowerPoint slideshows, etc.)

3. Students use email to contact experts and communicate with peers about a specific area ofinterest in order to expand knowledge.

Level 2 Students use some technology nearly every day in relation to learning activities. The primaryemphasis of technology use is still in the acquisition of basic skills, but there is evidence that thestudent is beginning to use technology to engage in learning practices that lead to new ways ofthinking, understanding, constructing knowledge and communicating results.

The vast majority of student technology use is teacher-directed, with students assigned to useparticular technology tools, applications, or resources as part of their academic work.

Level 2Evidence

As with Level 1, except technology use becomes more routine and regular. More advanced usesbegin. For example…

1. Students use a word processor or appropriate software to enhance the organization ofproducts, and to improve quality (i.e., cut-paste, spell check, grammar checking, etc)

2. Students use technology to solve problems (i.e., a graphing calculator, spreadsheet, database,etc.)

3. Students use technology to improve problem solving, reasoning, and thinking.4. Students use application programs to discover concepts and relationships, especially in

science, math, and social studies.5. Students use the Internet (download and copy, know useful web sites, and use search

engines) and appropriate technologies, such as CD-ROM encyclopedias, as research tools.

Level 1 The student’s initial use of technology supports the acquisition of basic skills and increasedproductivity.

All uses of technology are teacher-directed. Teachers weave the use of some technology toolsand devices into student lessons and activities.

Level 1Evidence

1. Students use calculators, games, tutorial, and application programs to practice and acquirebasic skills and to increase the depth of understanding of particular subject area curriculum.

2. Students use word processors to write (i.e., cut-paste, spell check, grammar checking, etc) andproduce products.

3. Students demonstrate the knowledge of basic application/productivity software such asspreadsheets, databases, and presentation tools.

4. Students begin to explore the Internet and become acquainted with email.

Rubric Tips

Even number of levelsCreate from ideal to lesser Yes, Yes But, No But, NoBe as descriptive as possibleDo not get excessively quantitative

Take a break...

FCPS Rubric Scores

0

0.5

1

1.5

2

2.5

3

3.5

4

Indicators

Student Impact

Teacher Fluency

Teacher Use withStudentsTeacherProductivityAdministratorSupportInfrastructureAllocation

Student Impact, 2

Technology is being used daily as a basic tool within a range of traditional learning activities

Some evidence that technology is supporting new ways of thinking and learning

Most student technology use is very teacher-directed and occurs in settings outside of the classroom

Teacher Fluency

Teacher Fluency = 2.5 Teachers are reasonably fluent, but need to

do more to connect technology to the core curriculum

Use of Technology with Students = 2.25 Teachers direct students to use technology,

but need to use technology to empower students as independent, engaged, learners

Teacher Productivity = 4 Teachers are very proficient in using

technology as a tool for personal and professional productivity

Administrator Support = 2 Administrators use technology, but need to

advocate and support more integrated uses among their teaching staff

Infrastructure Allocation, 2.75

The basic infrastructure -- hardware, network, software -- is complete and adequate

Individuals in Technology Support functions (schools and district) are doing an exemplary job and should receive much of the credit for the advances we have seen thus far

Schools need even more technology resource teacher support

Efforts must continue to map technology on to the core curriculum

Sum It All Up?

Fayette County is doing very well! Based on our research nationwide, FCPS is

clearly above average in most areas of its technology implementation

Teachers are using technology in ways that are consistent with their overall instructional approach Directed instruction Technology as a “skill” versus an integrated

element of the learning process

Technology can do more, and teachers can do more Professional development and increased

levels of resource teacher support will make this happen

The full report contains a number of recommendations for professional development and policy designed to positively impact performance on the indicators

To Summarize...

Start with your planFrom your goals, develop indicators and a

performance rubricAccumulate data and/or develop your data

collection tools How will you collect the data necessary to

document your stage of fulfillment?

Data Collection

Review Existing Data Current technology plan Curriculum District/school improvement plans

www.sun-associates.com/eval/sampleCreate a checklist for data collection

Collect New Data Focus Groups/Interviews

TeachersParentsStudentsAdministratorsOther stakeholders

Classroom Observations

Surveys Creating good surveys — what do you want to

know• length• differentiation (teachers, staff, parents, community, etc..)• quantitative data• attitudinal data• timing/response rates (getting returns!)

www.sun-associates.com/eval/samples/samplesurv.html

Solicitation of teacher/parent/student stories This is a way to gather truly qualitative data What does the community say about the use

and impact of technology?

Compile the reportDetermine how to share the report

School committee presentation Press releases Community meetings

Conclusion

Build evaluation into your technology planning effort

Remember, not all evaluation is quantitative

You cannot evaluate what you are not looking for, so it’s important to —

Develop expectations of what constitutes good technology integration

Technology and Traditional Assessment Methods

Does technology use result in higher test scores?

Probably, but there is not a direct linkTechnology alone is not helpful in

improving student performanceA high degree of technology integration is

more than likely to be an indicator itself of high performing schools and students

Is There a Connection?

High scores on technology integration rubrics generally equate to successful teaching and learning practices -- regardless of whether these practices have any specific relation to technology e.g., Constructivism, Engaged Learning, and

Student-Centered Classrooms

Technology supports these practices and these practices are shown to improve student learning

Schools where these practices are in place and technology is used appropriately are high performing schools

It’s not the technology, it’s the teaching!

We measure how well the technology is used to support successful teaching practices.

Our formative evaluations highlight how to improve teaching, technology integration, and ultimately student success

Next Steps?

Awareness building Establish the need for evaluation Establish a process

Division leadership As with technology implementation in general,

this is very important The evaluation will only be as good as it is

taken seriously

Evaluation committee membership Stakeholders!

Are tech plans in shape? Ideally, the evaluations will be based in sound

visions and goals and will take into account existing data on completed actions

Do you divisions have this capacity?

Evaluation work can be used to rejuvenate the local revision process, but this will take time and organization

Training It takes about a day to train people in this

approach/process Regional workshops? District teams?

Standardizing and cross-division review People should share and review their

questions, indicators, and rubrics before setting out to conduct an evaluation

Think about data collection Integrating a meaningful technology impact

survey into existing survey efforts Integrating observation data into existing

teacher/class observations

More Information

jsun@sun-associates.com 978-453-3070

www.sun-associates.com/eval/sample This presentation is linked to that page