Conducting Scientifically-Based Research in Teaching with

Technology, Part IISITE Annual Meeting Symposium

Atlanta, GeorgiaGerald Knezek & Rhonda Christensen

University of North TexasCharlotte Owens & Dale MagounUniversity of Louisiana at Monroe

March 2, 2004

Scientifically-Based Research(Whose Definition?)

• Methodology Issues• Randomization Issues• Instrumentation Issues• Determination of Impact• Analysis/Interpretation Issues

Issues of Methodology

• Quantitative – Currently in favor, heavy on analysis methodology

• Qualitative– Rich analysis, takes longer

• Mixed Methods– Seeing process in operation often necessary to find out

‘why’ in education

• Theory Building vs. Theory Testing• Exploratory/Data Mining vs. Hypothesis Testing

Issues of Randomization

• Random assignment (currently emphasized)– For internal validity (fidelity of experiment)– Start with large group– Randomly assign 1/2 treatment, 1/2 control

(Versus)• Random sampling

– Drawing from larger population– For generalizability to larger population– External validity (Trust that this would work elsewhere)– Also very important

Issues of Instrumentation

• Much emphasis on standardized outcome measures as ultimate (valid) criteria

• Less attention to reliability/accuracy of legislated tests and measures

• Little attention to how/where/when (or numerous other holes in) the data gathered

• Mistrust of teacher self appraisal/reflection

Analysis of Impact

• Hypothesis testing– Is the impact real (not due to chance?)– P = .05, .01, or none (vs. confidence intervals?)

• Effect size as a yard stick of impact

• Report p - level and ES

What is Effect Size?

• “… it is convenient to use the phrase ‘effect size’ to mean ‘the degree to which the phenomenon is present in the population,’ or ‘the degree to which the null hypothesis is false.” (Cohen, 1977, p. 9)

• Effect size is a standardized measure of the strength (degree of impact) of a discriminating feature or intervention.

How to Interpret Effect Size• Cohen’s d (1965, 1977, 1988) vs. other• Small (.2), medium (.5) vs. large (.8)

.2: IQ difference between twins vs. non-twins height difference between 15 and 16 yr. old girls

.5: large enough to be visible to naked eyeheight difference between 14 and 18 yr. Old girls

.8: IQ difference between college freshman and Ph.D.sheight difference between 13 and 18 yr. Old girls

• Compare to other common effect sizes– “As a quick rule of thumb, an effect size of 0.30 or greater is

considered to be important in studies of educational programs.” (NCREL)

– For example .1 is one month learning (NCREL)

SRI International. http://www.ncrel.org/tech/claims/measure.html

Effect Sizes are Known for Many Interventions (Ex: Dede’s Optimal Areas of Information Acquisition, 1990)

1. Learners construct knowledge rather than passively ingest information:Acceleration (Study 1) 1.00Acceleration (Study 2) .57Individualized Instruction (Study 1) .32Individualized Instruction (Study 2) A. Curriculum compacting .83 B. Credit for prior learning .5

2. Sophisticated information-gathering tools are used to stimulate the learner to focus on testing hypotheses rather than on plotting data:

Higher Order Questions(Study 1) .34Cognitive Processing (Study 3) .69

3. There is collaborative interaction with peers, similar to team-based approaches underlying today's science (Note that in constructivist methodology the teacher is considered a peer):

Reinforcement (Study 1) 1.17One to One Mentoring (Study 2) .57Social Skills (Study 3) .47

Hancock, R. J. (2003). The Expanded Will, Skill, Tool Model: A Step Toward Developing Technology Tools That Work. Paper presented to EdMedia 2003, Honolulu, Hawaii.

Issues of Analysis/Interpretation

• Much attention to single ‘correct’ procedure– T-test of differences vs. Analysis of Covariance– Power estimates for hierarchically nested data

• Little recognition of value of multiple views of data– Nonparametric techniques for small samples

• Too much emphasis on accept/reject null and too little on strength of effect (ES/APA)

• Tendency to use no data to make decisions rather than rely on less than perfect information

It’s all About ConfidenceAs shown in Figure 1, three of the measures’ 95% confidence intervals

… are roughly 3/4 of a confidence interval band above … that is, no more than 1/4 of the 95% confidence interval range overlaps from the upper to the lower group. Differences in this range are as a rule-of-thumb “meaningful” according to emerging APA guidelines, and roughly comparable to a p = .05 level of significance (Cumming, 2003). The effect size for the combined upper three versus the lower two is approximately [((3.09+3.05+2.95)/3) – ((2.32+2.41)/2]/ ((1.34+1.33+1.40+1.00+1.05)/5) = (3.03 – 2.37) / 1.22 = .66 / 1.22 = .54, considerably larger than the .30 cutoff beyond which technology interventions are considered meaningful (Bialo & Sivin-Kachala, 1996). Teachers rated the ARTS to the Delta class as much more useful for promoting interest in music and creating a positive effect on students’ overall education experience that for improving reading and math skills.

We’ve Long Been in the Credible Evidence Business

• Research based on dissertation criteria

• Quantitative to tell us what is happening; Qualitative to tell us why it is happening

KIDS Project: Technology Innovation Challenge Grant• KIDS - Key Instructional Design Strategies• 9.2 million 1999-2004 TICG• Replication of successful model to 50 rural school

districts• Major Project Components

– Technology Integration Professional Development– Technology-enhanced Reading Instruction

• External Evaluator: Univ. of North Texas• Research Agenda added to Project Evaluation

1999 - 2002 Findings

• High integration teachers make a critical difference for students without computers at home.

• Elementary school girls have equal or higher attitudes toward computers than boys.

• Technology skills of high school students are often higher than their teachers.

• Rural teachers and students fall between Allen ISD and Laredo on baseline measures of technology proficiency.

• Technology-infused reading activities accounted for approximately 10% of reading achievement gains.

Teacher Stage of Adoption vs. Home Access 2001(SITE 2003 Research Award)

Stages and Home Computer Access

0

5

10

15

20

25

30

35

Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Stage 6

No Home Comp

Home Comp

Trends in Computer Enjoyment - 2001 sample (Girls & Computers, NECC 2003)

Computer Enjoyment 2001

3.25

3.3

3.35

3.4

3.45

3.5

3.55

Grade 4 Grade 5 Grade 6

Male

Female

Linear (Female)

2002-2003 Experimental Design

• 7 randomly selected control districts• Compared with 18 treatment districts• Interventions:

– Summer Institute (Eisenhower Model)– Tools to integrate into the classroom– New technology-enhanced reading program

Technology Self Efficacy Gain(Pre-Post, 40-hour Summer Inst.)

Table 1.Summer 2002 KIDS Professional Development Institute Pre-Post Gains in Technology Proficiency(General Curriculum/Technology Integration Cohort, n = 109)

Pretest Mean Post Mean Pretest SD MeanGain Effect SizeTP-Email 4.19 4.52 0.80 0.33 0.41TP-WWW 4.16 4.72 0.86 0.56 0.65TP-IA 3.42 4.25 1.16 0.83 0.72TP-TT 3.61 4.53 1.16 0.92 0.79Note: Effect size = (post test rating in skill area – pretest rating in skill area) / pretest standard deviation.

Technology Self Efficacy Gain(Pre-Post, 40-hour Summer Inst.)

Technology Proficiency Self-Assessment by Ropp, 19992002 KIDS Institute

1

1.5

2

2.5

3

3.5

4

4.5

5

TP-Email TP-WWW TP-IA TP-TT

Pre

Post

TP Email TP-IA = Integrated Applications TP-WWW TP-TT = Te aching with Technology

Figure 1. Technology pre-post gains for 109 teachers enrolled in KIDS summer institute 2001 (Generalcurriculum/technology integration emphasis).

Is this good?

• “In keeping with American Psychological Association reporting guidelines (APA, 2001, p. 25), and in accordance with standards established by other scholarly sources, the indications of this generalized approach to reporting gains in terms of standard deviation units are that the summer 2002 professional development institute had a moderately large effect on the technology integration skills of teachers (Cohen, 1969) and resulted in gains well beyond the .3 benchmark commonly regarded as indicative of educational significance (Bialo, 1996)”.

2002-2003 Student Achievement Findings

• Treatment gained more than controls (p<.05)– Reading accuracy - Grade 1 and 2– Reading comprehension - Grade 2

• Average Effect Size = .33 (range = .23 - .89)• Many reading programs were used in typical

classrooms and some were very effective.• Students of teachers attending Institute gained

more (2 times ES).

Grade 1 vs. 2 Reading GainsTable 3.Texas Primary Reading Inventory First Grade Pre-Post Group Gain Statistics, Treatment vs. Control.

Measure Treatment/Control

n Mean SD T df One-TailSig

EffectSize

Treatment 210 4.87 2.33 .25Word List GainControl 100 4.29 2.24

2.11 202 .018

Treatment 203 1.23 .84 .50ReadingAccuracy Gain Control 99 .80 .87

4.06 189 .0005

Treatment 177 .86 .65 .23Story Level GainControl 93 .73 .45

1.89 250 .03

Treatment 238 .47 1.01 .05ExplicitQuestion Gain Control 119 .42 .91

.52 260 .30

Treatment 238 .32 .89 .10Implicit QuestionGain Control 118 .24 .70

.90 .288 .19

Note: Effect size = (post test rating in skill area – pretest rating in skill area) / pooled standard deviation.

Table 4.Texas Primary Reading Inventory Second Grade Pre-Post Group Gain Statistics, Treatment vs. Control.

Measure Treatment/Control

n Mean SD T df One-TailSig

EffectSize ofKIDS

Treatment 207 .59 .668 .42ReadingAccuracy Gain Control 50 .34 .52

2.49 255 .007

Treatment 242 .56 1.02 .24ExplicitQuestion Gain Control 70 .33 .90

1.73 310 .042

Treatment 242 .51 .86 1.77 310 .039 .23Implicit QuestionGain Control 70 .30 .95

Note: Effect size = (post test rating in skill area – pretest rating in skill area) / pooled standard deviation.

Research Design Template

• Evaluation Planned/Required:– Annual report to Dept. of Ed, 5-year summative

• Research Question 1:– Is the KIDS Summer Inst. effective in promoting

technology integration among teachers?

• Research Question 2:– Is there a positive impact of the KIDS technology-

based reading program on student achievement?

Research Design Template (cont.)

• Dependent Variable(s):– Gains in Level of Technology Integration (teachers)– Reading Achievement Gains (Grade 1-3 Students)

• Independent Variables:– Teachers: Before vs. after training– Students: Treatment vs. Control

classrooms/schools/districts

• Randomization Possible/Control Group:– Yes, 7 districts randomly selected from pool of 150

matching treatment group membership criteria

Instrumentation

• For Teachers:– Several instruments for attitudes, skills, and level of

technology integration capability– Reliabilities range from .78 to .95 for typical teachers

• For Students:– Texas Primary Reading Inventory (TPRI)– Used by more than 90% of Texas districts in K-2– Reliability reported as ‘high’ by creators– Story discontinuity and 2nd grade ceiling effect reported

by Knezek, Christensen & Dunn-Rankin (2003).

Data Analysis

• T-tests of treatment vs. control gain scores– Reading accuracy– Reading comprehension

• Analyis of covariance not carried out due to violations of assumptions

• Effect Size computations carried out using Cohen’s d (in spite of paired data)

Outlet for Findings

• SITE 2004 ;-)

• PT3 Leadership Institute Presentations

• Electronic Newsletter/Web Site

• 4th Annual Book on Project Findings nearing completion

Significance/Implications of Findings

• Technology - enhanced reading can be effective in promoting higher achievement in 1st and 2nd grade students

• The average effect size is “educationally meaningful”, on the order of an additional 3 months of achievement gain over 1 school year.

• The model has demonstrated “strong” evidence of effectiveness

Has KIDS Established “Strong Evidence” of Effectiveness?

For Additional Information

• View KIDS Project findings at http://iittl.unt.edu

• Contact: Gknezek@tenet.edu or

• RhondaC@tenet.edu