Assessment and Evaluation Feedback to Teams

Summer 2015

Data Collected

• GLE Multiple Choice Pre/Post• Essays (Interdisciplinary, Systems)• Summative Assessments• Attitude and Career Interest Pre/Post• Demographic data

GLE Pre/Post Composite Results

1 2 3 4 5 6 7 8 9 10 11 120

10

20

30

40

50

60

70

80

90

100GLE Data F12-S13

GLE Score

Num

ber o

f Stu

dent

s

Pre Course

Post Course

Pre 6.0 +/- 2.0Post 6.7 +/- 1.9Norm Gain = 11%n=424

1 2 3 4 5 6 7 8 9 10 11 120

20

40

60

80

100

120

140

160

180

200

GLE ScoreN

umbe

r of S

tude

nts

Pre

Post

Pre 6.8 +/- 1.9Post 7.0 +/- 2.0Norm Gain = 3%n=890

GLE Data F14-S15

Higher startAll in

GeoscienceCourses

~50%Geoscience

Courses

GLE Pre/Post

n=65n=39n=72

n=21n=21n=35

(n=72; Norm Gain = 15%) (n= 65, Norm Gain = 11%)

Interdisciplinary Essay

• Knowledge of Earth system interactions can influence how people make decisions about global challenges. Identify and describe a global challenge that society will likely face in the next 50 years. Explain how the science related to that challenge informs economic, social, and/or political decision making related to the global challenge you described.

Interdisciplinary Essay

• Geoscience Courses F12-S13• Range of topics InTeGrate

n=209

Climate Change WaterPopulation Fossil FuelsNon-renewables Sea LevelVolcano FoodNatural Disasters SoilsUnspecified Ozone LayerPollution Energy non-specificMultiple TsunamiDeforestation EarthquakeExtinctions HurricanesIce Melting

Akronn=153

Climate Change WaterPopulation Fossil FuelsNon-renewables Sea LevelVolcano FoodNatural Disasters SoilsUnspecified Ozone LayerPollution Energy non-specificMultiple TsunamiDeforestation EarthquakeExtinctions HurricanesIce Melting

Interdisciplinary Essay• Non Integrate similar F12-S13• Answers link to content

Interdisciplinary Essay

Complex SystemA systems thinker can identify a system (a natural system, a human system, a linked human/environment system), understand how that system can be divided into interacting parts, and recognize that changes in one part of the system will affect other parts of the system.

1. Give an example of a real-world system and describe its parts.2. Explain how parts of the system interact. Use systems concepts in your

explanation (e.g., positive and negative feedbacks, equilibrium, rates, etc.)

3. Using your example system, discuss how an effect in one part of that system can be influenced by multiple causal factors.

(Very) Preliminary Systems Essay Results (2014-15)

• Overview (n=168 answers)– Most students could identify a system with its components– 57% identified a “Geo” system; 43% “Non-Geo”

• Scores– 65% score 0 or 1– 33% score 2 (meant they wrote something about feedback)– 1 score 3

• Big Picture = Students are not getting this concept although the question does not appear to be the problem

10

Evaluation of assessments/student work

• Needed a way to document learning across the project

• Assessments are diverse– Relate to discrete content related goals– Relate to InTeGrate Guiding Principles– Methodology differed from year 1 to year 2– Year 1, feedback may be helpful during revisions– Year 2, feedback should be addressed

Rubric

• Evaluated extent to which collected assessments addressed module/unit goals– Can the goal be assessed with this artifact?

• Evaluated extent to which student work reflected progress toward meeting those goals– To what extent does the student work reflect

progress toward goal?• Same for InTeGrate guiding principles

Module Goal Rubric Elements

• 3 = Assessment/work clearly addresses major elements of goal

• 2 = Assessment/work lacks one or more major elements

• 1 = Assessment/work missing multiple key elements

• 0 = Assessment/work does not appear to address goals

Introduction to the InTeGrate Evaluation Program

Inputs Activities OutcomesOutputs

The Evaluation Team

Frances LawrenzEduc. PsychologyU. Of Minnesota

Ellen IversonSERC

Carlton College

Carol BaldassariProgram Evaluation

Research GroupEndicott College

Kim KastensColumbia University

herehere

Vali MaraCenter for International Earth Science Information Network

15

What we told NSF we would do(1) Evaluate the project’s effectiveness at “expanding the number of students who enroll….”

(2) Evaluate the project’s effectiveness at achieving “…enhanced learning…”

(3) Evaluate the project’s effectiveness at achieving “… significant progress towards addressing the national challenge” of environmental sustainability

(4) Formatively evaluate the partnership

(5) Evaluate the effectiveness of the InTeGrate implementation program model

16

Tools and approaches• Database of participants in all workshops, meetings,

teams, courses, etc. • Pre-/post-instruction measures of students, probing

attitudes, geoscience literacy, and ability to tackle grand challenges

• Reflection surveys of materials developers and testers at key points throughout development/testing process

• Interviews with selected leaders and participants• Document examination: websites, instructional

materials, etc. • Observation of selected meetings, workshops, etc. • Mapping of flows of influence and information (logic

model) 17

http://www.pbs.org/wgbh/nova/sciencenow/3410/03-ever-01.html

Evaluation examines the effectiveness of InTeGrate in the aggregate, and does not monitor individual students, teachers, or modules. We focus on the flock, not the birds.

18

What you can expect

• Spreadsheet of GLE Pre- Post- answers by team (Emailed at a later date)

• Spreadsheet of Attitude Pre- Post- answers by team (Emailed at a later date)

• Evaluation of assessments and student work

19

How to use this feedback• GLE/Attitude – analyze as you wish. Since

numbers are small, analysis has limited utility• Essays – Consider overall quality of answers

and how they do or do not relate to module• Evaluation of assessments

– Address as warranted during revisions

20