CRITICAL EVALUATION OF CRITICAL EVALUATION OF EDUCATIONAL RESEARCHEDUCATIONAL RESEARCH

IntroductionIntroduction

Quantitative ResearchQuantitative Research

Qualitative ResearchQualitative Research

Quiz Quiz

Why is it important to learn to critically Why is it important to learn to critically evaluate educational research?evaluate educational research?

• To help determine the effectiveness of new (and old) educational practices;

• To distinguish between well-designed and poorly designed research studies;

• To participate in / evaluate data-based decision-making (those that have the data control the agenda).

Has anyone done any research?Has anyone done any research?

QuantitativeQuantitative ResearchResearchanalysis of numerical data

QualitativeQualitative ResearchResearchanalysis of data such as words, pictures, or objects

Now picture this….Now picture this….

Up next, Quantitative Research….Up next, Quantitative Research….

Dan Meyer says: Dan Meyer says:

1. Use multimedia;

2. Encourage student intuition;

3. Ask the shortest question you can;

4. Let students build the problem;

5. Be less helpful.

(Design a sequel.)

17. A clear, octagonal tank measures 12 inches across, has sides of length 5 inches and is 14 inches tall. It is being filled with water at the rate of 12 oz per 5 seconds. Assume that this rate does not vary over the course of the filling process. How long will it take to fill the tank?

Who doesn’t like a good math word problem now and then?

Is Dan Right?Is Dan Right?

Deliberate/Non-Deliberate Bias or Distortion of Research

U.S. Department of Education Official:

“3 year old children of professional mothers have larger working vocabularies than “welfare mothers” of 3 year olds.”

Google that!Google that!

Be a wise consumer of educational research….Be a wise consumer of educational research….

Some of the following slides are adapted from a presentation by Dr. J. Anderson.

Actual research finding:

“When 3 year olds of professional mothers talk to their mothers, they use more words than do mothers on welfare when talking to their own 3 year olds. Welfare mothers use far larger vocabularies when talking to their friends, their older children, or the researchers themselves.”

Hare, B. & Risley, T. (1995). Meaningful differences. Baltimore; Paul H. Brookes.

Research Question

The research question determines the focus of a research study. The remainder of the study is a systematic process of obtaining an answer to that question.

The purpose of the research question is to tell the reader the intent of the study and set the stage for what follows.

Nuts & BoltsNuts & Bolts

Examples:1. Do boys and girls in coed and same-sex

classes differ in regard to their perceptions of student behavior, teacher support, and competitiveness?

2. Is there an association between high school grades and study time?

3. What is the impact of playing classical music in the classroom on students’ academic achievement in mathematics?

Research Hypothesis

• The research hypothesis is a predictive statement about the expected outcome of the research

• The research hypothesis dictates the method and design of the research and gives the reader an idea of what the design should look like.

Nuts & BoltsNuts & Bolts

Example:“Children who attend preschool will make

greater gains in first grade reading achievement than comparable children who do not attend preschool.”

The Independent VariableThe Independent VariableIs the presumed cause of some

educational phenomena

The Dependent VariableThe Dependent VariableIs the presumed effect of some

educational phenomena

Nuts & BoltsNuts & Bolts

Examples

1. State mandated testing of all students will have a negative effect on teacher morale.

2. High teacher morale results in lower teacher absenteeism than low teacher morale.

3. Having the teacher shake hands with each student as they enter the classroom will increase student active participation in class discussion.

Experimental GroupGroup that receives the independent (treatment)

variable

Comparison Group (Control, Contrast)Has the same characteristics as the experimental group

but does not experience the independent variable.

SamplePortion of a population that is included in the study.

PopulationTotal number of people or things from which a smaller

sample is drawn.

Nuts & BoltsNuts & Bolts

Measures of Central TendencyMeasures of Central Tendency

Mean (M)

This is an average but we think of it as “the” average. It is calculated by dividing the sum of the scores by the number of scores.

Median

The middle score in a distribution of scores.

Mode

The most frequently occurring score in the distribution.

Nuts & BoltsNuts & Bolts

Measures of Variability or SpreadMeasures of Variability or Spread

Standard Deviation (SD)Standard Deviation (SD)

Is a measure of the extent to which, scores in a distribution, on the average, deviate from their mean. How “spread out” are the data?

RangeRange

Is the distance between the lowest and highest scores in a distribution.

Nuts & BoltsNuts & Bolts

Statistical SignificanceStatistical Significance

A statistical test (T test, chi square, ANOVA) may be conducted to determine the probability that the difference between mean scores of two groups can be attributed to chance.

Probability value = p

Traditional levels of significance .05, .01, .001

ns = not significantNuts & BoltsNuts & Bolts

Don’t confuse the level of significance (p value) with the practical educational significance of the research results.

The level of significance is strongly influenced by the number of individuals (n) included in the research project.

1,000 subjects; M1=42, M2=47 p<.01

50 subjects; M1=42, M2=52 ns

Statistical SignificanceStatistical Significance

Nuts & BoltsNuts & Bolts

K. p<.012

L. p<.041

• p<.10

• p<.100

• p<.051

• p<.067

• p<.367

• p<.35

• p<.20

• p<.50

A. p<.01

B. p<.05

C. p<.00

D. p<.000

E. p<.02

F. p<.024

G. p<.17

H. p<.111

I. p<.283

J. p<.83

Group A Group B

Pre 38 Pre 34

Post 43 Post 42

p<.05 p<.01

True or False:It is more likely that the difference between the

mean pre and post test scores of Group A is due to chance than it is that the difference between the mean pre and post test scores of Group B is due to chance.

ValidityValidity

Refers to the degree to which scientific observations actually measure what they purport to measure.

Nuts & BoltsNuts & Bolts

ReliabilityReliability

Refers to the repeatability of scientific observations. It is the consistency with which the data measure what we are attempting to measure over time.

Nuts & BoltsNuts & Bolts

GeneralizabilityGeneralizability

Refers to the applicability of research findings to settings and contexts different from the one in which they were obtained.

Nuts & BoltsNuts & Bolts

Finally, A good research document empowers readers to reach their own conclusions by including:

• A well-defined question;

• Description of the context and existing information about an issue;

• Consideration of various perspectives;

• Presentation of evidence, with data and analysis in a format that can be replicated by others;

• Discussion of critical assumptions, contrary findings, and alternative interpretations;

• Cautious conclusions and discussion of their implications;

• Adequate references, including original sources, alternative perspectives, and criticism.

Evaluating Research Quality - Guidelines for Scholarship - 22 February 2012

By Todd Litman - Victoria Transport Policy Institute

And really finally, a negative exemplar…

Bad Research

Consume research responsibly

The Dangers of Bread

The following is a humorous example of how legitimate-sounding statistics can be applied with false logic to support absurd arguments.

http://www.geoffmetcalf.com/bread.html

“A recent headline read, ‘Smell of baked bread may be health hazard.’ The article described the dangers of harmful air emissions from baking bread. I was horrified. When are we going to do something about bread-induced pollution? Sure, we attack tobacco companies, but when is the government going to go after Big Bread? Well, I’ve done a little research, and what I’ve discovered should make anyone think twice...”

The Dangers of Bread

1. More than 98% of convicted felons are bread eaters.

2. Fully half of all children who grow up in bread-consuming households score below average on standardized tests.

3. More than 90% of violent crimes are committed within 24 hours of eating bread.

5. Bread is made from a substance called “dough.” It has been proven that as little as one pound of dough can suffocate a mouse.

8. Primitive tribal societies that have no bread exhibit a low occurrence of cancer, Alzheimer’s, Parkinson’s disease and osteoporosis.

-8. Bread is baked at temperatures as high as 400o F! That kind of heat can kill an adult in less than one minute.