Sample Outcome Data from Makes Sense Strategies Research Impact on WRITING Impact on VOCABULARY Impact on INFORMATION PROCESSING SKILLS Impact on CONTENT

Sample Outcome Data from Makes Sense Strategies Research

Impact on WRITING

Impact on VOCABULARY

Impact on INFORMATION PROCESSING SKILLS

Impact on CONTENT LEARNING

Impact on CLASSROOM TEST SCORES

MAKES SENSE STRATEGIES MAKES SENSE STRATEGIES 4 Teaching & Learning4 Teaching & Learning


The study compared …Graphic Organizers depicting basic organization structures

vs. Essential Understanding Smart-sheets

vs.Generative Idea

Smart-sheets

The study measured…Breadth, depth, & accuracy of knowledge of 98 students

Impact on learning when different kinds of cues are embedded into graphic organizers

STUDY 1STUDY 1

16 high achieving students16 typical achieving students16 low achieving students16 low achieving students w/LD


TEXT RESOURCESCLASS

1

GRAPHIC ORGANIZERS depicting basic ORGANIZATION STRUCTURES

SMART-SHEETS featuring

GENERATIVE IDEAS about a topic

PHASE 1 PHASE 2 PHASE 3 PHASE 4

Teacher employed a different strategy during Teacher employed a different strategy during each phase of instructioneach phase of instruction

STUDY 1STUDY 1

SMART-SHEETS featuring ESSENTIAL

UNDERSTANDINGS about a topic


Pre & Post measures of students’ Pre & Post measures of students’

Breadth of knowledgeBreadth of knowledge

Depth of knowledgeDepth of knowledge

Accuracy of knowledgeAccuracy of knowledge

TEXT RESOURCESCLASS

1





STUDY 1STUDY 1



Learning measured for each phaseLearning measured for each phase


TEXT RESOURCESCLASS

1





STUDY 1STUDY 1



2nd teacher used same strategies, but 2nd teacher used same strategies, but in a different orderin a different order

CLASS 2



TEXT RESOURCES





TEXT RESOURCESCLASS

1





STUDY 1STUDY 1



CLASS 2



TEXT RESOURCES






TEXT RESOURCES




CLASS 3






TEXT RESOURCES

CLASS 4

3rd & 4th teachers used same 3rd & 4th teachers used same strategies, also in different ordersstrategies, also in different orders

MAKES SENSE STRATEGIES MAKES SENSE STRATEGIES 4 Teaching & Learning4 Teaching & Learning STUDY 1STUDY 1RESULTSRESULTS

2-way analysis of variance to analyze interaction effects

GROUP X TREATMENT

Data analyzed using…

Tukey’s Post-hoc analysis of significant interactions

MAKES SENSE STRATEGIES MAKES SENSE STRATEGIES 4 Teaching & Learning4 Teaching & Learning STUDY 1STUDY 1

Relative impact of strategy on knowledge gainsRelative impact of strategy on knowledge gains

RESULTSRESULTS

ACCURACY OF KNOWLEDGEACCURACY OF KNOWLEDGE

Same pattern found, regardless of type of studentSame pattern found, regardless of type of student high achieving

typical achieving

low achieving

low achieving w/LD


Text Text ResourcesResources

SMART-SHEETS featuring GENERATIVE IDEAS about a topic

SMART-SHEETS featuring ESSENTIAL UNDERSTANDINGS about a topic

NONO statistically significant differences found statistically significant differences found between each instructional approach on between each instructional approach on students’ students’ accuracyaccuracy of knowledge learned of knowledge learned

In other words, each treatment was equally effective (or not effective) for the different types of students on measures of accuracy


Relative impact of strategy on knowledge gainsRelative impact of strategy on knowledge gainsDEPTH OF KNOWLEDGEDEPTH OF KNOWLEDGE

RESULTSRESULTS







DEPTH OF KNOWLEDGEDEPTH OF KNOWLEDGE

Again, each of type of Again, each of type of student responded similarly student responded similarly

to each type of instructionto each type of instruction

RESULTSRESULTS

Significant differences found Significant differences found between each instructional between each instructional approach on students’ approach on students’ DEPTHDEPTH of knowledge of knowledge








RESULTSRESULTS

Essential Essential UnderstandingsUnderstandings

Most beneficial to ALL Most beneficial to ALL groups of studentsgroups of students








RESULTSRESULTS

Basic Graphic Basic Graphic Organizers pretty Organizers pretty effective as welleffective as well


Relative impact of strategy on knowledge gainsRelative impact of strategy on knowledge gainsBREADTH OF KNOWLEDGEBREADTH OF KNOWLEDGE

RESULTSRESULTS







BREADTH OF KNOWLEDGEBREADTH OF KNOWLEDGE

RESULTSRESULTS

Significant differences found Significant differences found between each instructional between each instructional approach on students’ approach on students’ BREADTHBREADTH of knowledge of knowledge

Again, each of type of Again, each of type of student responded similarly student responded similarly

to each type of instructionto each type of instruction







BREADTH OF KNOWLEDGEBREADTH OF KNOWLEDGE

RESULTSRESULTS

Essential Essential UnderstandingsUnderstandings

CLEARLY most CLEARLY most beneficialbeneficial

MAKES SENSE STRATEGIES MAKES SENSE STRATEGIES 4 Teaching & Learning4 Teaching & Learning STUDY 1STUDY 1RESULTSRESULTS

So how well do the different treatments reduce the achievement gap b/w typical learners & students w/LD?

Impact of Graphic Organizers & Essential Understanding Smart-sheets on high school students’ BREADTH of relational knowledge via social studies instruction

21% less

TA

HA

LALD

29% less

27% less

Traditional Text Materials High achievers mean level of BREADTH of knowledge

Gap between high & typical achievers

Gap between typical achievers & students w/ LD

21% less

TA

HA

LALD

29% less

27% less

TA

HA

LALD


21% less

TA

HA

LALD

29% less

27% less

Traditional Text Materials

Graphic Organizer Materials

TA

HA

LALD


21% less

TA

HA

LALD

29% less

27% less



TA

HA

LALD

4 % less

11 % less

9 % less 16 % pt reduction in

gap


25 % less

21% less

TA

HA

LALD

29% less

27% less



TA

HA

LALD

4 % less

Essential Understanding Smart-sheers

17 % less

HA

LALD

TA

9 % less 11 % less

13 % less

TA

HA

LALD

HA

Impact of Graphic Organizers & Essential Understanding Smart-sheets on high school students’ DEPTH of relational knowledge via social studies instruction

21% less

TA

LALD

29% less

27% less


TA

HA

LALD

21% less

TA

HA

LALD

29% less

27% less

TA

HA

LALD


21% less

TA

HA

LALD

29% less

27% less



TA

HA

LALD


21% less

TA

HA

LALD

29% less

27% less



TA

HA

LALD

2 % less

20 % less 19 % less

TA

HA

LALD


11% less

21% less

TA

HA

LALD

29% less

27% less



TA

HA

LALD

2 % less

Essential Understanding Smart-sheers

16 % less

HA

LALD

TA20 % less

11% less

19 % less

1st place

Teachers overwhelmingly preferred because… …Helped them focus on what was important

…Made planning lessons easier

…Resulted in more structured lessons & discussions

…Liked how student elaboration was prompted, thereby making learning of history more applicable to students’ lives today and prompting students to practice higher order thinking.

…Liked variety of uses and adaptability of the graphic organizers with topic-specific prompts across many classroom situations.

…Perceive them as are of benefit to all students regardless of ability level or disability status.

Most favored by teachers and students


Essential understandings Smart-sheets

STUDY 1STUDY 1RESULTSRESULTS

STUDENTS overwhelmingly preferred because…

… Helped them focus on what was important

… Found this type to be interesting

… Valued the opportunities to elaborate

… Perceived that they enhanced their learning & studying

Most favored by teachers and students


GOs containing topic-specific “essential understanding” prompts

STUDY 1STUDY 1RESULTSRESULTS

5th Grade Inclusive Classroom

How well do what teachers say were key ideas and important details they hoped students learned match with what students actually

reveal were learned?

MSS impact on teachers’ ability to successfully identify and teach relevant main ideas and important details

to students w/LD


Before MSS After MSS

0 4 Ave. number of main ideas

about the lesson identified by students that matched with main ideas the teacher identified that she

intended students to learn

Ave. number of important facts

(or details) students identified by students that matched with facts the teacher identified

that she intended students to learn

2.4 12

5th grade students were interviewed immediately following social studies lesson. Interviewer webbed information as students identified ideas and directed interviewer where to note them on the web.

1. What was that whole lesson about?

2. What are some really important main ideas or different topics from the lesson? Why do you think they are important?

3. What are some specific things or facts from the lesson you think were really important? Why do you think they were important?



Impact on composition writing of 8th grade students with learning disabilities

Typical 8th grader

N=20

117 words

CONTROL group of students w/LD

N = 28

EXPERIMENTAL group of students w/LD N = 28

FLUENCY

Pretest Post-test Pretest Post-testTypical 8th grader produces an average of 117 words on high-stakes essays

8th students with LD typically produce about 27 words on the same task

Experimental Group Students w/LD29

words26

words

Control Group Students w/LD

STUDY 3STUDY 3



Typical 8th grader

N=20

117 words


EXPERIMENTAL group of students w/LD

FLUENCY

Pretest Post-test Pretest Post-test

Experimental Group Students w/LD29

words26

words

Control Group Students w/LD

On the post test, the control group of students w/LD produced slightly fewer words than they did on the pretest

24 words

STUDY 3STUDY 3



Typical 8th grader

N=20

117 words


EXPERIMENTAL group of students w/LD

FLUENCY

Pretest Post-test Pretest Post-test

29 words

26 words

24 words

Experimental group of students w/LD (received MSS instruction)

126words

+97 words more than pretest

+9 words more than typical 8th grader

STUDY 3STUDY 3

9 Schools Extremely low

performance

8 Schools Low performance

7 Schools Moderate performance

2 Schools Good performance


High-Stakes Writing Assessment

This study examined the impact of MSS on schools with varying degrees of prior performance

This type of study shows what kind of school is likely to benefit the most from MSS

STUDY 4STUDY 4

Following 1 year MSS Following 1 year MSS implementation implementation


performance




21.6%before MSS

38.83%before MSS

58.39%before MSS

73.02%before MSS

MSS

62.41%

23.08 pt. gain

MSS

74.81%

15.67 pt. gain

MSS

49.24%

27.67 pt. gain

MSS

82.68%

9.79 pt. gain

Lower-performing schools definitely demonstrate the most dramatic improvements



STUDY 4STUDY 4


performance


MSS

62.41%


MSS

74.81%


MSS

49.24%

MSS

82.68%

21.6%before MSS

38.83%before MSS

58.39%before MSS

73.02%before MSS

23.08 pt. gain 15.67 pt. gain27.67 pt. gain 9.79 pt. gain

Moderate-performing schools closed the gap to within 9% pts

of Good-performing schools



STUDY 4STUDY 4



performance


MSS

62.41%


MSS

74.81%


MSS

49.24%

MSS

82.68%

21.6%before MSS

38.83%before MSS

58.39%before MSS

73.02%before MSS


MSS school gains when compared to the state-wide average



STUDY 4STUDY 4

State-wide average = .05 pt. gain during same period


7th grade English Language Learners / ESL

2002-03 2003-04 Gains

24% 75%

51 pts

Makes Sense Strategies implementation

California Writing Assessment% students meeting standards

STUDY 5STUDY 5

State Ave 30.90%

01-02 40.8%

02-03 57.39%

03-04

MSS #1 36.1%

28.08%

51.43%

MSS

implementation 1ST YEAR

23.35 pt gain!

16.59 pt gain


Alabama Writing Assessment (High-stakes) Elementary 5th grades (semi-rural)

% students meeting or exceeding standards

State Ave 30.90%

01-02 40.8%

02-03 57.39%

03-04 59.0%

04-05

MSS #1 36.1%

28.08%

51.43% 81.0%

MSS

implementation 1ST YEAR 2nd YEAR

29.57 pt gain!

1.61 pt gain

STUDY 6STUDY 6MAKES SENSE STRATEGIES MAKES SENSE STRATEGIES 4 Teaching & Learning4 Teaching & Learning STUDY 6STUDY 6



State Ave 30.90%

01-02 40.8%

02-03 57.39%

03-04 59.0%

04-05

2nd YEAR

81.0%

MSS #1 36.1%

28.08%

51.43%

MSS

implementation 1ST YEAR




State Ave 30.90%

01-02 40.8%

02-03 57.39%

03-04 59.0%

04-05

MSS #1 36.1%

28.08%

51.43% 81.0%

MSS

implementation

MSS #2 22.0%

10.0% 62.24%

1ST YEAR 2nd YEAR

52.24 pt gain!

71.0%8.76

pt gain




State Ave 30.90%

01-02 40.8%

02-03 57.39%

03-04 59.0%

04-05

MSS #1 36.1%

28.08%

51.43% 81.0%

MSS

implementation

MSS #2 22.0%

10.0% 62.24%

1ST YEAR 2nd YEAR

71.0%




MSS #3 11.2%

15.09%

63.63%

State Ave 30.90%

01-02 40.8%

02-03 57.39%

03-04 59.0%

04-05

MSS #1 36.1%

28.08%

51.43% 81.0%

MSS

implementation

MSS #2 22.0%

10.0% 62.24% 71.0%

1ST YEAR 2nd YEAR

48.54 pt gain!




34.0%MSS #3 11.2%

15.09%

63.63%

State Ave 30.90%

01-02 40.8%

02-03 57.39%

03-04 59.0%

04-05

MSS #1 36.1%

28.08%

51.43% 81.0%

MSS

implementation

MSS #2 22.0%

10.0% 62.24% 71.0%

1ST YEAR 2nd YEAR

29.63 pt LOSS!




22.4%34.0%MSS #3 11.2%

15.09%

63.63%

State Ave 30.90%

01-02 40.8%

02-03 57.39%

03-04 59.0%

04-05

MSS #1 36.1%

28.08%

51.43% 81.0%

MSS

implementation

MSS #2 22.0%

10.0% 62.24% 71.0%

1ST YEAR 2nd YEAR

29.63 pt LOSS!

Why do you think this score dropped so drastically?

School thought they had “made it” in writing so emphasis switched to other issues





22.4%34.0%MSS #3 11.2%

15.09%

63.63%

State Ave 30.90%

01-02 40.8%

02-03 57.39%

03-04 59.0%

04-05

MSS #1 36.1%

28.08%

51.43% 81.0%

MSS

implementation

MSS #2 22.0%

10.0% 62.24% 71.0%

1ST YEAR 2nd YEAR

29.63 pt LOSS!Leadership & support for

using MSS disappeared!

Leadership expectations for fidelity & support must be on-going to ensure sustainability!

IMPLICATIONS


7th Grade Alabama Writing Assessment

2002-03 2003-04 Gains

Hi-performing school Comparison school 1

41.62% -2.17%43.79%

Comparison school 2

28.50% -7.05%35.55%

Makes Sense school

37.65% 60.51% 22.86%

92.64% 85.08% -6.66%

7th Grade Alabama Writing Assessment 2002-03 2003-04 Gains

Hi-performing school Comparison school 1Comparison school 2

41.62%28.50%

-2.17%-7.05%

43.79%35.55%

Makes Sense school

37.65% 60.51% 22.86%

92.64% 85.08% -6.66%


15.56%3.13%

-3.79%-3.39%

19.35%6.52%

Makes Sense school

14.21% 20.00% 5.79%

64.00% 65.00% 1%


27.50%23.40%

-7.19%-10.07%

34.69%35.55%

Makes Sense school

27.08% 43.83% 20.75%

NA NA NA

High School 10th grades (semi-rural) % students meeting or exceeding standards


53%

03-04

04-05

MSS High

School77% 26 pt gain!

2 pt gain

Non-MSS High School

MSS implementation

51%

51%

Both groups performed at the same levels in 2003-04

Action Research by SchoolsAlabama Writing Assessment

% students meeting standards 2002-03 2003-04 Gains

Differentiating

Making information more learnable

Using brain-based pedagogy

curriculum

32.71%

57.84%

25.13%

Rural 7th GradeSuburb 7th Grade

38% 61% 23.00%

96%

POST

84%

POST

85%

PRE

Control group of 6th grade typical achievers not receiving MSS

84%

PRE

Experimental group of 6th grade typical achievers receiving MSS

54%

PRE

Experimental group of 6th grade students w/LD receiving MSS

12 pt gain

87%

POST

33 pt gain

MAKES SENSE STRATEGIES MAKES SENSE STRATEGIES 4 Teaching & Learning4 Teaching & Learning STUDY 9STUDY 9Impact on vocabulary test scores 6th grade inclusion class

Concept test scores of students with LD

Before After Gains 79

%92%93%

14 pts30 pts30 pts

High Soc Economic 6th grade

62%63%

65%Mid Soc Economic 4/5th grade

Low Soc Economic 4/5th grade


97%

POST

81%

PRE

11TH grade typical achievers

71%

PRE

11th grade students w/LD

17 pt gain 26 pt gain

98%

POST


Impact on concept test scores

11th grade inclusion social studies class

Documents

Sample Outcome Data from Makes Sense Strategies Research Impact on WRITING Impact on VOCABULARY Impact on INFORMATION PROCESSING SKILLS Impact on CONTENT