This three year project was funded in part by the National Science Foundation (NSF) REESE # 0635559 and the Minnesota Pollution Control Agency (MPCA) EE grant. For further information, visit the Best Practices for Field Days web site, http://www.extension.umn.edu/fielddays/ Papers available upon request from Stephan P. Carlson at [email protected] B E S T P R A C T I C E S F O R Field Days Field Days Field Days Field Days Using Observation to Measure the Impacts of Environmental Field Days What does it take to develop a valid and reliable Observation Tool for assessing Field Day programs? 1 IDENTIFYING BPFD COMPONENTS 3 ESTABLISHING CODER RELIABILITY 4 VALIDATING THE MEASURE 2 DEVELOPMENT OF ASSESSMENT TOOL, PROTOCOL and TRAINING 5 APPLICATION AND IMPACT Authors: Stephan P. Carlson PhD, Martin Storksdieck PhD, Joe E. Heimlich PhD, Nathan J. Meyer, and Andrea Lorek Strauss ■ Employed Modified Delphi study to identify components using WebCT ■ Twenty-seven experts nationwide were involved in the Delphi ■ Subgroup of 10 finalized the subcomponents and observable items ■ Results were 2 measurement tools; an Individual Tool to measure learning stations, and a Holistic Tool to measure the full day experience ■ Drafted Tools, Rubric and Protocol ■ Established 4-5 hour observation training with video coding ■ Trained 30 observers on using the Tools and building concensus ■ Tested the Tool at 8 Field Day programs statewide ■ Five observers coded each learning station ■ Focus Group of coders established usability and effectiveness of the Tool ■ Kappa Test calculated interrater reliability ■ Tested the observation measure against a student survey designed to measure the same items ■ Correlation of observer assessment and student survey proved positive ■ 2008 Children’s Water Festival compared 16 observers to 16 classrooms ■ 20 of 26 items correlated between the two measures This tool meets the “Psychometrics of Direct Observation” by J. Hintze (2005) in the measurement of behavior. As an “absolute measure” it is sound, reliable and has face, construct and content validity. Needs: ➤ Over 76% of Minnesota’s 87 counties offer Environmental Field Day programs. (Carlson, 2008) ➤ Field Day programs reach over 15,000 fourth-sixth grade youth yearly. (Carlson, 2008) ➤ MN Extension Educators in Natural Resources identified Field Days as a key area in their strategic plan in 2000. ➤ BPFD developed a Program Planning Guidebook for Organizers, Presenters, Teachers and Volunteers (Meyer et al., 2005) K=∑ i=n i=1 (d i )/n with d i = Distance for individual observation score on each item and n= the number of observers.

Using Observation to Measure the Impacts of Environmental

Download PDF Report

Upload
others
View
3
Download
0

Embed Size (px)

Citation preview

Page 1: Using Observation to Measure the Impacts of Environmental

This three year project was funded in part by the National Science Foundation (NSF) REESE # 0635559 andthe Minnesota Pollution Control Agency (MPCA) EE grant.

For further information, visit the Best Practices for Field Days web site, http://www.extension.umn.edu/fielddays/Papers available upon request from Stephan P. Carlson at [email protected]

B E S T P R A C T I C E S F O R

Field DaysField DaysField DaysField DaysUsing Observation to Measure the Impacts of Environmental Field Days

What does it take to develop a valid and reliable Observation Tool for assessing Field Day programs?

1 IDENTIFYING BPFD COMPONENTS

3 ESTABLISHING CODER RELIABILITY

4 VALIDATING THE MEASURE

2 DEVELOPMENT OF ASSESSMENT TOOL, PROTOCOL and TRAINING

5 APPLICATION AND IMPACT

Authors: Stephan P. Carlson PhD, Martin Storksdieck PhD, Joe E. Heimlich PhD, Nathan J. Meyer, and Andrea Lorek Strauss

� Employed Modified Delphi study to identify components using WebCT

� Twenty-seven experts nationwide were involved in the Delphi

� Subgroup of 10 finalized the subcomponents and observable items

� Results were 2 measurement tools; an Individual Tool to measure learningstations, and a Holistic Tool to measure the full day experience

� Drafted Tools, Rubric and Protocol

� Established 4-5 hour observation training with video coding

� Trained 30 observers on using the Tools and building concensus

� Tested the Tool at 8 Field Day programs statewide

� Five observers coded each learning station

� Focus Group of coders established usability and effectiveness of the Tool

� Kappa Test calculated interrater reliability

� Tested the observation measure against a student survey designed to measure the same items� Correlation of observer assessment and student survey proved positive

� 2008 Children’s Water Festival compared 16 observers to 16 classrooms� 20 of 26 items correlated between the two measures

This tool meets the “Psychometrics of Direct Observation” by J. Hintze (2005) in the measurement ofbehavior. As an “absolute measure” it is sound, reliable and has face, construct and content validity.

Needs:� Over 76% of Minnesota’s 87 counties offer Environmental Field Day programs. (Carlson, 2008)� Field Day programs reach over 15,000 fourth-sixth grade youth yearly. (Carlson, 2008)� MN Extension Educators in Natural Resources identified Field Days as a key area in their strategic plan in 2000.� BPFD developed a Program Planning Guidebook for Organizers, Presenters, Teachers and Volunteers (Meyer et al., 2005)

K=∑i=ni=1(di)/n

with di= Distance for individual observation score on each item and n= the number of observers.