0Evaluating Success: HIVE Playable Technology Project
January 13, 2016
Our Vision is for All Children to ExcelMISSION: Transform the academic achievements, self-confidence, and life trajectories of children living in under-resourced communities. APPROACH: BELL partners with schools and community organizations to combine resources and leverages time in the summer and after school to improve student achievement.15School Partners3,500Scholars Served (2015)
Summer Learning Loss
66% of the academic achievement gap between disadvantaged students and their higher-income peers in 8th grade can be traced to unequalaccess to summer learning opportunities.
+1 MonthMiddle-Income Students Gain Academic Skills in the Summer-2 MonthsDisadvantaged Students Lose Academic Skills in the SummerReading Achievement, Middle - Income StudentReading Achievement, Disadvantaged Student
Source: National Summer Learning Association
When not engaged in structured learning activities in the summer, children from low-income families experience summer learning loss of 2 months grade-equivalent reading and math skills.Only 25% of students attend summer learning programs, and in many urban communities as few as 5% of at-risk students attend.Summer learning losses over time account for 66% of the academic achievement gap by the time a student completes the 8th grade.Summer learning is also directly linked to whether students attend college preparatory classes, graduate high school on time, and attend four-year colleges.
Evaluation at BELL
Why Measure?Short term impact on students basic math and literacy skillsEffectiveness of curriculumEffectiveness of teachers and program staffEngagement of parents/guardians and students with learningWhat students like and don't likeHow to improve
Assessment & EvaluationQUANTITATIVE TOOLSSTAR Enterprise AssessmentsScholar Management System (Salesforce.com)Independent EvidenceQUALITATIVE TOOLSParent SurveyTeacher SurveyQuality Monitoring System
A rigorous, independent evaluation of BELL Summer found strong evidence that the program increases scholars academic skills and parental engagement.
Challenges: Measuring Success of Playable Technology Program How to test students for improvement in specific curricular concepts? For example, problem solving or implementing hypothesis.
How to gauge students engagement with STEM enrichment program?
1. BELL measures total outcomes in math and literacy calculated in months gained and overall levels of interest in the BELL program2. Therefore, data is not collected on any specific program3. Leaving us unable to understand the effectiveness of any particular program enhancements specific curricular concepts 4. For example, the playable technology program helped students design and implement wearable tech and intentionally utilized a cross section of subject areas and concepts - examples: problem solving and hypothesis implementation
Solutions: CONTROL GROUP COMPARISON
ENHANCE CURRENT MEASURING TOOLS
ARRANGE CIRRICULUM GOALS AROUND PRE-DETERMINED CONCEPTS
Sample Updated Survey:
* Underperforming scholars score in the lowest quartile on STAR assessments at the start of the summer: Urgent Intervention (1-10th percentile) or Intervention (11-25th percentile)* Underperforming scholars score in the lowest quartile on STAR assessments at the start of the summer: Urgent Intervention (1-10th percentile) or Intervention (11-25th percentile)* Underperforming scholars score in the lowest quartile on STAR assessments at the start of the summer: Urgent Intervention (1-10th percentile) or Intervention (11-25th percentile)
Enhanced Curriculum :The Scientific Method
We plan to adapt the curriculum to reflect the scientific method. The scientific method provides clear guidelines for instructors to gauge student comprehension of basic curricular concepts that are not captured through the STAR assessment tool.