The SoI Pilot National Evaluation: Findings and Recommendations 2013-06-27آ  The SoI Pilot National

  • View
    0

  • Download
    0

Embed Size (px)

Text of The SoI Pilot National Evaluation: Findings and Recommendations 2013-06-27آ  The SoI Pilot...

  • The SoI Pilot National

    Evaluation: Findings

    and Recommendations

    NASA’s Summer of Innovation Project - Updated

    Cambridge, MA

    Bethesda, MD Task Order Durham, NC NNH10CC70D Under Atlanta, GA BPA NNH08CD70Z

    Revised February 14, 2011

    Prepared for National Aeronautics and Space

    Administration (NASA)

    Office of Education

    Prepared by Abt Associates, Inc.

    Hilary Rhodes

    Kristen Neishi

    Melissa Velez

    Jackie Mendez

    Alina Martinez

    Education Development Center, Inc.

    Jackie DeLisi

    Abt Associates Inc.

    55 Wheeler Street

    Cambridge, MA 02138-1168

  • Abt Associates Inc. i

    Acknowledgements

    This report is the collaborative effort of numerous individuals. This study was conducted with the

    assistance of Abigail Levy, Yueming Jia, and Linda Hirsch of the Education Development Center,

    Inc.; Diana Epstein, Kimberly Francis, Beth Gamse, Amanda Parsad, and Jan Nicholson of Abt

    Associates Inc.; Estella Sena, Kevin McMahon, and Robb Magaw of Abt-SRBI; Ryoko Yamaguchi

    of the SERVE Center, the University of North Carolina at Greensboro; and Eric Hagedorn of the

    University of Texas, El Paso.

    We gratefully acknowledge the cooperation and assistance received from the SoI sites in providing

    reporting forms, collecting consent forms, administering surveys, and sharing their experiences with

    us. The SoI national evaluation team would also like to thank the students and educators for their

    contributions in completing surveys.

    We also appreciate the guidance and support of staff from the NASA Office of Education, including

    Brian Yoder who served as the program officer for the evaluation, as well as Dovie Lacy, Katie

    Pruzan, Rick Gilmore, Carol Galica, Kristina Brink, and Shelley Canright.

    Lastly, we also thank the reviewers of this report for the helpful feedback provided.

    The data collection, analysis and reporting of this material was conducted in accordance with

    OMB Control No. 2700-0142 (expiration July 30, 2011).

  • Abt Associates Inc. ii

    Contents

    Executive Summary ............................................................................................................................. 1

    Chapter 1: Overview of the SoI Pilot Project .................................................................................... 3 Introduction.................................................................................................................................. 3

    Vision for the SoI Pilot Project .................................................................................................... 3

    FY 2010 SoI Pilot Implementation .............................................................................................. 5

    Organization of the Report........................................................................................................... 6

    Definition of Key Terms.............................................................................................................. 6

    Chapter 2. SoI Pilot National Evaluation........................................................................................... 7 Objectives for the SoI Pilot National Evaluation......................................................................... 7

    Design of the SoI Pilot National Evaluation................................................................................ 8

    SoI Pilot National Evaluation’s Data Collection ....................................................................... 11

    Analytic Approach ..................................................................................................................... 12

    Limitations to the Evaluation’s Approach ................................................................................. 12

    Chapter 3. SoI Pilot Activities ........................................................................................................... 13 Sites’ Goals & Objectives .......................................................................................................... 13

    SoI Pilot Funding ....................................................................................................................... 15

    Management of SoI Activities ................................................................................................... 16

    Site’s Program Models............................................................................................................... 19

    Plans for Post-Summer 2010 Implementation ........................................................................... 22

    Chapter 4. SoI Pilot Survey Results.................................................................................................. 24 Survey Response Rates .............................................................................................................. 24

    Survey Analysis ......................................................................................................................... 25

    Participant Characteristics.......................................................................................................... 26

    Survey Results ........................................................................................................................... 34

    Potential Areas for Future SoI Professional Development ........................................................ 42

    Conclusions................................................................................................................................ 44

    Limitations to the SoI Pilot National Evaluation Surveys ......................................................... 45

    Chapter 5. SoI Pilot Successes........................................................................................................... 46 Extensive, National Reach ......................................................................................................... 46

    Engaging Student Activities....................................................................................................... 47

    Exposure to NASA Content and Materials ................................................................................ 48

    Developing Relationships between Educators and University Faculty ..................................... 49

    Chapter 6. SoI Pilot Implementation Challenges ............................................................................ 50 Time Frame................................................................................................................................ 50

    Lack of Clarity about SoI Priorities ........................................................................................... 53

    Engaging Underserved and Underrepresented Adolescents ...................................................... 54

    Logistics..................................................................................................................................... 54

  • Abt Associates Inc. iii

    Chapter 7. SoI Pilot National Evaluation’s Lessons Learned ........................................................ 56 Communication.......................................................................................................................... 56

    Data Collection Process ............................................................................................................. 59

    Content of Collected Data.......................................................................................................... 61

    Chapter 8. Conclusions and Recommendations .............................................................................. 63 SoI Project.................................................................................................................................. 63

    SoI Sites ..................................................................................................................................... 67

    National Evaluation Team ......................................................................................................... 68

    References ........................................................................................................................................... 72

    Appendix A. Planning Reporting Form ........................................................................................... 73

    Appendix B. Implementation Reporting Form................................................................................ 79

    Appendix C. “Technical Assistance” Call Agenda.......................................................................... 82

    Appendix D. Post-Implementation Debrief Protocol ...................................................................... 84

    Appendix E. Baseline Student Survey .............................................................................................. 86

    Appendix F. Follow-Up Student Survey........................................................................................... 95

    Appendix G. Baseline Educator Survey ........................................................................................... 99

    Appendix H. Educator Follow-Up Survey ..................................................................................... 107

    Appendix I. Teacher Survey Justification..................................