Studying the Impact of Online Professional Development on Educators’ Technology Integration Skills

AERA Annual Meeting (2005, Montreal) 1

Studying the Impact of Online Studying the Impact of Online Professional Development on Professional Development on

Educators’ Technology Educators’ Technology Integration SkillsIntegration Skills

Tom Clark, PhDTom Clark, PhDTA ConsultingTA Consulting

Elizabeth Oyer, PhDElizabeth Oyer, PhDEvaluation SolutionsEvaluation Solutions

In In 63.03063.030 Questions, Methods and Findings from Program Evaluations Questions, Methods and Findings from Program Evaluations and Research Examining Digital Online Learning Projects. Le Centre and Research Examining Digital Online Learning Projects. Le Centre

Sheraton Montreal, Salon 4, 4:05-5:35 pm, April 14, 2005Sheraton Montreal, Salon 4, 4:05-5:35 pm, April 14, 2005

2 AERA Annual Meeting (2005, Montreal)

Consulting Consulting Evaluation SolutionsEvaluation Solutions

Study Context Study Context

• STAR Project: TechKnowledgy InitiativeSTAR Project: TechKnowledgy Initiative– PURPOSE: Increase tech integration PURPOSE: Increase tech integration

preparedness of K-12 teachers via online PDpreparedness of K-12 teachers via online PD– Created in 1997, became centerpiece of 5-year Created in 1997, became centerpiece of 5-year

STAR Project in 2000STAR Project in 2000– Participants follow a reflective PD model using Participants follow a reflective PD model using

a Star-Online VTLC www.star-online.org/tka Star-Online VTLC www.star-online.org/tk



Study Context Study Context

• Leaders In Technology Enhanced Schools (LITES) Kit Leaders In Technology Enhanced Schools (LITES) Kit ProjectProject– PURPOSE: Increase tech integration in inquiry units of K-12 PURPOSE: Increase tech integration in inquiry units of K-12

teachersteachers– Used packaged inquiry curriculum integrated with technology in a Used packaged inquiry curriculum integrated with technology in a

portable format portable format

• Common evaluation questionCommon evaluation question– Does project participation increase the quality and level of Does project participation increase the quality and level of

teachers’ technology integration?teachers’ technology integration?

• Methodological issueMethodological issue– Is there a valid and reliable way to measure changes in quality Is there a valid and reliable way to measure changes in quality

and level of tech integration?and level of tech integration?



MethodsMethods• ParticipantsParticipants

– 30 teachers (60 online and video cases total)30 teachers (60 online and video cases total)– 27 teachers for face-to-face ratings 27 teachers for face-to-face ratings

• Data SourcesData Sources– Video of STAR units implemented in classroomVideo of STAR units implemented in classroom– Online artifacts for STAR portfoliosOnline artifacts for STAR portfolios– FTF observation of LITES teachersFTF observation of LITES teachers



MethodsMethods• InstrumentInstrument

– MTimms Technology Observation Instrument MTimms Technology Observation Instrument originally developed for FTF observationoriginally developed for FTF observation

– Addresses style of teaching, types of tech use, Addresses style of teaching, types of tech use, tech integrationtech integration

• DesignDesign– Initial study: subscales combined for composite Initial study: subscales combined for composite

score; correlations provide reliability estimatesscore; correlations provide reliability estimates– Follow-up study: subscales interpreted in Follow-up study: subscales interpreted in

original scale; inter-rater consensus computedoriginal scale; inter-rater consensus computed



ResultsResults

• Correlations between raters in initial studyCorrelations between raters in initial study– rrVideoVideo(36)=.735, p<.01(36)=.735, p<.01

– rrOnlineOnline(36)=.684, p<.01(36)=.684, p<.01



ResultsResults

• Consensus Across MethodsConsensus Across Methods

% Inter-Rater Consensus All Methods

0 25 50 75 100

FTF

Video

Online



ResultsResults

• Consensus for Video RatingsConsensus for Video Ratings

% Inter-Rater Consensus - Video

0 25 50 75 100

InteractionStudent Autonomy

General Level of IntegrationTech Integration w/Topic

Tech ComplexityTeacher Tech UseStudent Tech Use

Classroom ManagementStudent Role

Student EngagementProp of Stud Using Tech



ResultsResults

• Consensus for Online RatingsConsensus for Online Ratings

% Inter-Rater Consensus - Online

0 25 50 75 100

Cognitive Activity

Interaction

Student Autonomy

General Level of Integration

Tech Integration w/Topic

Tech Complexity

Teacher Tech Use

Student Tech Use



DiscussionDiscussion

• Reliability of tool not yet fully acceptableReliability of tool not yet fully acceptable

• Positive results for face-to-face ratings Positive results for face-to-face ratings highlight potentialhighlight potential

• Post-analysisPost-analysis– Individual rater biasIndividual rater bias– Shorter rating periodShorter rating period



ImplicationsImplications

• If reliability and validity established, several If reliability and validity established, several research questions of interest. For STAR:research questions of interest. For STAR:– What is the correspondence between teacher What is the correspondence between teacher

plans for tech integration (through plans for tech integration (through portfolios/artifacts) and actual integration?portfolios/artifacts) and actual integration?

– How do tech proficiency and level of How do tech proficiency and level of experience influence this relationship?experience influence this relationship?



Thank You!Thank You!

Tom Clark, PhD Tom Clark, PhD

TA Consulting TA Consulting

[email protected]@yahoo.com

(217)585-1539(217)585-1539

Elizabeth Oyer, PhD Elizabeth Oyer, PhD

Evaluation SolutionsEvaluation Solutions

[email protected]@evalsolutions.com

(317) 582-1925(317) 582-1925

Documents

Studying the Impact of Online Professional Development on Educators’ Technology Integration Skills