Click here to load reader

Best Practices for Evaluating Mobile Learning: Real ... 2018 Mobile Learning Eval... · PDF file Benefits of mobile learning Elementary school principals Middle school principals

  • View

  • Download

Embed Size (px)

Text of Best Practices for Evaluating Mobile Learning: Real ... 2018 Mobile Learning Eval... ·...

  • Best Practices for Evaluating

    Mobile Learning:

    Real Stories from Around the Globe

    Dr. Julie A. Evans CEO, Project Tomorrow June 25, 2018 @SpeakUpEd @JulieEvans_PT

  • Dr. Julie A. Evans Chief Executive Officer, Project Tomorrow

    Chief Researcher, Speak Up Research Project

    Mobile Learning Project Evaluator since 2007

    Also, skilled in the evaluation of blended learning and digital content usage in K-12 and higher education

    Thank you for being here!

  • Special Guest:

    Hiram Lopez-Landin Senior Analyst, Wireless Reach Qualcomm

  • Today’s discussion topics

    ▪ About our work o Project Tomorrow and the Speak Up Research Project o Qualcomm Wireless Reach

    ▪ Why are evaluations of mobile learning important today?

    ▪ State of mobile learning today & evaluations for mobile learning

    ▪ Let’s talk about 3 different mobile learning projects

    ▪ My top 8 tips and strategies for evaluating mobile learning

    ▪ Extended learning – links to some additional resources

    ▪ Your thoughts, insights, questions

  • Tweet with us!

    @JulieEvans_PT @SpeakUpEd @hlopezlandin @QCWirelessReach

  • Getting to know you!

  • About Project Tomorrow (

    ▪ Nonprofit education organization supporting K-12 education since 1996

    ▪ Mission is to ensure all of today’s students are well prepared for the future

    ▪ Programs and research focus on role of digital tools within the education ecosystem – believe in power of STEAM to support student preparation for college and career success

    o Speak Up Research Project on Digital Learning: collecting & reporting on the authentic feedback of K-12 stakeholders to inform federal, state & local programs and policies

  • About the Speak Up Project (

    ▪ Annual research project since 2003

    ▪ Uses online surveys + focus groups

    ▪ Facilitated 100% through schools and districts

    ▪ We design online surveys to collect feedback from your K-12 Students, Teachers, Parents, Administrators, and Community Members

    ▪ All K-12 schools – public, private, parochial, charter, virtual - are eligible to participate

    ▪ Project Tomorrow manages all data collection and reporting for you - 100% free service

    ▪ Schools get summary report with all locally collected data + state and national data for benchmarks

    We share national data with federal, state and local policymakers to inform programs and funding

  • About the Speak Up Project (

    Survey question topics include:

    ✓ Use of technology to support learning ✓ School climate for innovation ✓ College and career ready skill development/interest ✓ Leadership challenges ✓ Teachers’ needs for professional learning ✓ How do different stakeholders value digital learning ✓ Emerging trends w/digital tools, content and resources ✓ New classroom models: mobile, blended, flipped ✓ School to home communications ✓ Designing the ultimate school

    ❖ Activities

    ❖ Attitudes

    ❖ Aspirations

    Since 2003, 5.4 million K-12 stakeholders have submitted a Speak Up survey

  • 1 0

    Qualcomm Wireless Reach

  • 11









    Wireless Reach Education Portfolio Impact to Date

  • 12









    United States

    East Asia

    Southeast Asia

    South Asia


    Latin America



    Programs by Region Percent of female beneficiaries in all education programs


    Organizations served in education portfolio570

    Wireless Reach Education 10 Years of Accomplishments

    Devices used in education portfolio11,000

    All figures include aggregated data from 2006 to 2017

  • 13

    Education Strategy

    Wireless Reach education

    programs are designed to address

    the barriers to adoption of wireless

    technology in the classroom

    including the need for:

    • Digital content & assessment

    • Infrastructure & IT support

    • Privacy & security

  • Why is it important to evaluate our mobile learning projects?

  • Why is it important to evaluate our mobile learning projects?

    Are our investments yielding a return on investment?

  • What do you see?

    Are we all seeing

    “evaluation” or even

    “mobile learning”

    the same way?

  • State of Mobile Learning: selected research findings from Speak Up 2017

    State of Mobile Learning: selected research findings from Speak Up 2017

    Learning on the go – anytime, anywhere

  • Survey Audience # of Surveys Submitted

    K-12 Students 340,927

    Teachers & Librarians 34,833

    Parents 23,159

    Administrators 3,249

    Community Members 4,611

    About schools and districts: 3,641 districts, 10,619 schools 29% urban, 34% rural, 37% suburban, 68% title 1 eligible

    National participation in Speak Up 2017: 406,779

  • @Project Tomorrow 2017

    Students have more personal access to mobile devices than ever before





    53% 57% 55%

    39% 32% 35%

    48% 54%

    10% 15% 14% 13%15% 12%

    K-2 Gr 3-5 Gr 6-8 Gr 9-12

    Smartphone Tablet Laptop Digital Reader Smartwatch

  • @Project Tomorrow 2017

    Teachers and mobile devices within learning

    Classroom access is changing rapidly

    In class access

    2015 2017

    BYOD 17% 23%

    1:1 in school assignment

    20% 31%

    1:1 in school + home assignment

    12% 14%

    Cart access 31% 30%

    No access 31% 18%

    Not surprising: teachers want

    more PD on how to

    effectively integrate mobile

    devices within instruction

  • @Project Tomorrow 2017

    Administrators say mobile learning = positive results

    ▪ 49% of school principals say they see positive academic results from their 1:1 mobile learning programs

    Benefits of mobile learning Elementary school


    Middle school principals

    High school principals

    Students develop topics more fully 82% 83% 77%

    Students are more engaged in learning 72% 62% 55%

    Preparation for college/work 71% 71% 71%

    Increased access to online content 69% 73% 73%

    Learning is more personalized 65% 60% 57%

    Students are collaborating more 57% 58% 58%

    Improved teacher-student communications 48% 58% 63%

  • But what metrics are school and district leaders using to measure outcomes from mobile learning?

  • Turn to a neighbor and share: How are you evaluating the success of your mobile learning programs?

    How satisfied are you with your evaluation process?

  • Audience feedback

  • Administrators say these are the top ways they are measuring the impact of their digital initiatives:

    Metrics % of district administrators


    Which ones are you using?

    Student engagement levels 77%

    Classroom observations 69%

    Student feedback 62%

    Student achievement results 60%

    Teacher buy-in 54%

    Teacher feedback 53%

    Parent feedback 37%

  • Two of the 8 essentials:

    ✓Purposeful planning for mobile device usage

    ✓Measuring project results with meaningful metrics

  • ✓Purposeful planning for mobile device usage

    ▪ What are your goals? ▪ What outcomes do

    you want to see? ▪ Why is this important

    for your school or district?

    ✓Measuring project results with meaningful metrics

    ▪ Aligning metrics with goals

    ▪ Leveraging appropriate tools

    ▪ Being open to what you don’t know yet

    Evaluating mobile learning begins with project planning

  • Introduction to 3 mobile learning projects, the metrics they used/are using to evaluate program success & lessons learned that can inform your efforts:

    ❑ “Making Learning Mobile” – Chicago

    ❑ “STEAMing Ahead with Mobile Learning – San Diego

    ❑ “Classroom of the Future” – Turkey

    Let’s get real!

  • “Making Learning Mobile” – Chicago

    o Three year focused study on the impact of extended Internet access and mobile learning on student achievement

    o Study group: 5th grade students and their teachers at Falconer Elementary School in Chicago, IL

    o Students each assigned a Samsung Android tablet with 4G LTE enabled education broadband to use at school and at home

    Partners: Kajeet, Qualcomm Wireless Reach, Project Tomorrow

  • “Making Learning Mobile” – Chicago

    Objectives of the project was to learn more about:

    ✓ How did the students and teachers use the