The Long and Winding Road: How the Integration of

Evaluation Performance Measures and Results Can

Lead to Better Quality Evaluations

Gale A. Mentzer, PhD

Director of Evaluation Services

The University of Toledo, Ohio

Required ObjectiveIncrease the number of K-12 teachers using educational technology themselves and with their students.

Original Logic ModelInputs Outputs Outcomes

Teacher Preparation Program: Course in Educational Technology

Minimum 3.0 on required technology course.

Students are trained in the use of a variety of educational technology tools.

Faculty Modeling of use of technology

Student reporting on frequency of faculty modeling.

All faculty model the integration of technology in teaching

Student practice through Methods and Student Teaching

Student reporting on confidence using specific technology tools 11 on 15 pt scale); on application of technology tools to student teaching (score of 11 on 15 pt. scale); review of student teaching portfolios show evidence of both planned and actual integration of technology.

Student are confident using technology tools, are confident using them to enhance instruction, and portifios show the use of technology during student teaching .

Actual ResultsInputs Outputs Findings Outcomes

Teacher Preparation Program:

Course in Educational Technology

Minimum 3.0 on required technology course.

Average grade 3.17. Students are trained in the use of a variety of educational technology tools.

Faculty Modeling of use of technology

Student reporting on frequency of faculty modeling.

Students report little faculty modeling of technology to enhance instruction.

Faculty not modeling technology - Are faculty tech-savvy?

Student practice through Methods and Student Teaching

Student reporting on confidence using specific technology tools; on application of technology tools to student teaching (score of 11 on 15 pt. scale); review of student teaching portfolios.

Average confidence score 10 with little confidence using content-specific tools; student report using technology during student teaching in a variety of innovative ways; student portfolios show no or rudimentary use of technology

Student are below expected confidence levels; students report the use of technology during student teaching but artifacts (portfolios) do not support this.

Faculty Modeling

• Explore faculty perceptions about the role of technology in their courses

• Expectations as to pre-service teacher use of technology

• Ask where in program is technology as an instructional tool taught

Outputs

• Purpose of technology education is to enhance instruction

• Modeling of technology as an instructional tool takes place in all courses

• Faculty consciously include assignments and activities that illustrate the use of technology to enhance instruction

• Faculty will emphasize the benefits of using technology when teaching

Findings

• Purpose of technology education is to provide competency

• Modeling of technology as an instructional tool takes place in Educational Technology and Methods courses only

• Faculty do not plan to use technology but rather do so when convenient to their purposes

• Faculty seldom or never share the benefits of using technology when teaching with their students

vs

Self-Reporting vs. Portfolios• Review official guidelines and

requirements for student teaching portfolios

• Examine student teaching portfolio performance assessment rubrics

• Ask faculty whether integrating technology as an educational tool is required of student teaching

Outputs

• Guide and requirements include evidence of using technology to enhance teaching

• Performance assessment rubrics include the evaluation of a technology-enhanced lesson

• Faculty agree that integrating technology as an instructional tool is part of student teaching experience

Findings

• Questions as to how technology might be integrated but no requirement that it actually be integrated

• Performance assessment does not include technology integration

• Faculty were split as to whether technology integration in student teaching was required or not

vs

Wait a minute. . .

UT3 Graduate Technology Survey

• Sent to all in-service teachers • Explore attitudes towards using

technology in teaching• Determine extent to which

teachers and their students use technology to teach and learn

Outputs

• Teachers will value the integration of technology into teaching.

• Most (80%) of teachers will use technology to provide instruction at least twice a week.

• Students will use technology to complete assignments or learn concepts at least once a month.

Findings

• 68% felt is was essential; 28% felt it was important (96% total)

• 94% of the teachers use technology at least twice a week to provide instruction (45% used it 4 or 5 days/week).

• 76% use technology at least once a month to complete assignments and 77% use it to learn concepts.

vs.

Summative Results

Summative Results