The SoI Pilot National Evaluation: Findings and Recommendations 2013-06-27آ  The SoI Pilot National

The SoI Pilot National

Evaluation: Findings

and Recommendations

NASA’s Summer of Innovation Project - Updated

Cambridge, MA

Bethesda, MD Task Order Durham, NC NNH10CC70D Under Atlanta, GA BPA NNH08CD70Z

Revised February 14, 2011

Prepared for National Aeronautics and Space

Administration (NASA)

Office of Education

Prepared by Abt Associates, Inc.

Hilary Rhodes

Kristen Neishi

Melissa Velez

Jackie Mendez

Alina Martinez

Education Development Center, Inc.

Jackie DeLisi

Abt Associates Inc.

55 Wheeler Street

Cambridge, MA 02138-1168

Abt Associates Inc. i

Acknowledgements

This report is the collaborative effort of numerous individuals. This study was conducted with the

assistance of Abigail Levy, Yueming Jia, and Linda Hirsch of the Education Development Center,

Inc.; Diana Epstein, Kimberly Francis, Beth Gamse, Amanda Parsad, and Jan Nicholson of Abt

Associates Inc.; Estella Sena, Kevin McMahon, and Robb Magaw of Abt-SRBI; Ryoko Yamaguchi

of the SERVE Center, the University of North Carolina at Greensboro; and Eric Hagedorn of the

University of Texas, El Paso.

We gratefully acknowledge the cooperation and assistance received from the SoI sites in providing

reporting forms, collecting consent forms, administering surveys, and sharing their experiences with

us. The SoI national evaluation team would also like to thank the students and educators for their

contributions in completing surveys.

We also appreciate the guidance and support of staff from the NASA Office of Education, including

Brian Yoder who served as the program officer for the evaluation, as well as Dovie Lacy, Katie

Pruzan, Rick Gilmore, Carol Galica, Kristina Brink, and Shelley Canright.

Lastly, we also thank the reviewers of this report for the helpful feedback provided.

The data collection, analysis and reporting of this material was conducted in accordance with

OMB Control No. 2700-0142 (expiration July 30, 2011).

Abt Associates Inc. ii

Contents

Executive Summary ............................................................................................................................. 1

Chapter 1: Overview of the SoI Pilot Project .................................................................................... 3 Introduction.................................................................................................................................. 3

Vision for the SoI Pilot Project .................................................................................................... 3

FY 2010 SoI Pilot Implementation .............................................................................................. 5

Organization of the Report........................................................................................................... 6

Definition of Key Terms.............................................................................................................. 6

Chapter 2. SoI Pilot National Evaluation........................................................................................... 7 Objectives for the SoI Pilot National Evaluation......................................................................... 7

Design of the SoI Pilot National Evaluation................................................................................ 8

SoI Pilot National Evaluation’s Data Collection ....................................................................... 11

Analytic Approach ..................................................................................................................... 12

Limitations to the Evaluation’s Approach ................................................................................. 12

Chapter 3. SoI Pilot Activities ........................................................................................................... 13 Sites’ Goals & Objectives .......................................................................................................... 13

SoI Pilot Funding ....................................................................................................................... 15

Management of SoI Activities ................................................................................................... 16

Site’s Program Models............................................................................................................... 19

Plans for Post-Summer 2010 Implementation ........................................................................... 22

Chapter 4. SoI Pilot Survey Results.................................................................................................. 24 Survey Response Rates .............................................................................................................. 24

Survey Analysis ......................................................................................................................... 25

Participant Characteristics.......................................................................................................... 26

Survey Results ........................................................................................................................... 34

Potential Areas for Future SoI Professional Development ........................................................ 42

Conclusions................................................................................................................................ 44

Limitations to the SoI Pilot National Evaluation Surveys ......................................................... 45

Chapter 5. SoI Pilot Successes........................................................................................................... 46 Extensive, National Reach ......................................................................................................... 46

Engaging Student Activities....................................................................................................... 47

Exposure to NASA Content and Materials ................................................................................ 48

Developing Relationships between Educators and University Faculty ..................................... 49

Chapter 6. SoI Pilot Implementation Challenges ............................................................................ 50 Time Frame................................................................................................................................ 50

Lack of Clarity about SoI Priorities ........................................................................................... 53

Engaging Underserved and Underrepresented Adolescents ...................................................... 54

Logistics..................................................................................................................................... 54

Abt Associates Inc. iii

Chapter 7. SoI Pilot National Evaluation’s Lessons Learned ........................................................ 56 Communication.......................................................................................................................... 56

Data Collection Process ............................................................................................................. 59

Content of Collected Data.......................................................................................................... 61

Chapter 8. Conclusions and Recommendations .............................................................................. 63 SoI Project.................................................................................................................................. 63

SoI Sites ..................................................................................................................................... 67

National Evaluation Team ......................................................................................................... 68

References ........................................................................................................................................... 72

Appendix A. Planning Reporting Form ........................................................................................... 73

Appendix B. Implementation Reporting Form................................................................................ 79

Appendix C. “Technical Assistance” Call Agenda.......................................................................... 82

Appendix D. Post-Implementation Debrief Protocol ...................................................................... 84

Appendix E. Baseline Student Survey .............................................................................................. 86

Appendix F. Follow-Up Student Survey........................................................................................... 95

Appendix G. Baseline Educator Survey ........................................................................................... 99

Appendix H. Educator Follow-Up Survey ..................................................................................... 107

Appendix I. Teacher Survey Justification..................................