Best Practices for Evaluating Mobile Learning: Real ... 2018 Mobile Learning Eval... · Benefits of mobile learning Elementary school principals Middle school principals High school

Best Practices for Evaluating

Mobile Learning:

Real Stories from Around the Globe

Dr. Julie A. EvansCEO, Project Tomorrow June 25, 2018 @SpeakUpEd@JulieEvans_PT

Dr. Julie A. EvansChief Executive Officer, Project Tomorrow

Chief Researcher, Speak Up Research Project

Mobile Learning Project Evaluator since 2007

Also, skilled in the evaluation of blended learning and digital content usage in K-12 and higher education

Thank you for being here!

Special Guest:

Hiram Lopez-LandinSenior Analyst, Wireless ReachQualcomm

Today’s discussion topics

▪ About our work o Project Tomorrow and the Speak Up Research Project o Qualcomm Wireless Reach

▪ Why are evaluations of mobile learning important today?

▪ State of mobile learning today & evaluations for mobile learning

▪ Let’s talk about 3 different mobile learning projects

▪ My top 8 tips and strategies for evaluating mobile learning

▪ Extended learning – links to some additional resources

▪ Your thoughts, insights, questions

Tweet with us!

@JulieEvans_PT @SpeakUpEd@hlopezlandin @QCWirelessReach

Getting to know you!

About Project Tomorrow (www.tomorrow.org)

▪ Nonprofit education organization supporting K-12 education since 1996

▪ Mission is to ensure all of today’s students are well prepared for the future

▪ Programs and research focus on role of digital tools within the education ecosystem – believe in power of STEAM to support student preparation for college and career success

o Speak Up Research Project on Digital Learning: collecting & reporting on the authentic feedback of K-12 stakeholders to inform federal, state & local programs and policies

About the Speak Up Project (www.tomorrow.org/speakup)

▪ Annual research project since 2003

▪ Uses online surveys + focus groups

▪ Facilitated 100% through schools and districts

▪ We design online surveys to collect feedback from your K-12 Students, Teachers, Parents, Administrators, and Community Members

▪ All K-12 schools – public, private, parochial, charter, virtual - are eligible to participate

▪ Project Tomorrow manages all data collection and reporting for you - 100% free service

▪ Schools get summary report with all locally collected data + state and national data for benchmarks

We share national data with federal, state and local policymakers to inform programs and funding

About the Speak Up Project (www.tomorrow.org/speakup)

Survey question topics include:

✓ Use of technology to support learning ✓ School climate for innovation ✓ College and career ready skill development/interest ✓ Leadership challenges ✓ Teachers’ needs for professional learning✓ How do different stakeholders value digital learning ✓ Emerging trends w/digital tools, content and resources ✓ New classroom models: mobile, blended, flipped✓ School to home communications✓ Designing the ultimate school

❖ Activities

❖ Attitudes

❖ Aspirations

Since 2003, 5.4 million K-12 stakeholders have submitted a Speak Up survey

10

Qualcomm Wireless Reach

11

45

PROGRAMS

17

COUNTRIES

150

STAKEHOLDERS

270K

PEOPLE

Wireless Reach Education Portfolio Impact to Date

12

17

9

7

6

2

2

1

1

United States

East Asia

Southeast Asia

South Asia

Middle…

Latin America

Sub-Saharan…

Europe

Programs by RegionPercent of female beneficiaries in all education programs

49%

Organizations served in education portfolio570

Wireless Reach Education 10 Years of Accomplishments

Devices used in education portfolio11,000

All figures include aggregated data from 2006 to 2017

13

Education Strategy

Wireless Reach education

programs are designed to address

the barriers to adoption of wireless

technology in the classroom

including the need for:

• Digital content & assessment

• Infrastructure & IT support

• Privacy & security

Why is it important to evaluate our mobile learning projects?

Why is it important to evaluate our mobile learning projects?

Are our investments yielding a return on investment?

What do you see?

Are we all seeing

“evaluation” or even

“mobile learning”

the same way?

State of Mobile Learning: selected research findings from Speak Up 2017

State of Mobile Learning: selected research findings from Speak Up 2017

Learning on the go – anytime, anywhere

Survey Audience # of Surveys Submitted

K-12 Students 340,927

Teachers & Librarians 34,833

Parents 23,159

Administrators 3,249

Community Members 4,611

About schools and districts: 3,641 districts, 10,619 schools29% urban, 34% rural, 37% suburban, 68% title 1 eligible

National participation in Speak Up 2017: 406,779

@Project Tomorrow 2017

Students have more personal access to mobile devices than ever before

35%

50%

79%

91%

53%57% 55%

39%32% 35%

48%54%

10%15% 14% 13%15% 12%

K-2 Gr 3-5 Gr 6-8 Gr 9-12

Smartphone Tablet Laptop Digital Reader Smartwatch


Teachers and mobile devices within learning

Classroom access is changing rapidly

In class access

2015 2017

BYOD 17% 23%

1:1 in school assignment

20% 31%

1:1 in school + home assignment

12% 14%

Cart access 31% 30%

No access 31% 18%

Not surprising: teachers want

more PD on how to

effectively integrate mobile

devices within instruction


Administrators say mobile learning = positive results

▪ 49% of school principals say they see positive academic results from their 1:1 mobile learning programs

Benefits of mobile learning Elementary school

principals

Middle school principals

High school principals

Students develop topics more fully 82% 83% 77%

Students are more engaged in learning 72% 62% 55%

Preparation for college/work 71% 71% 71%

Increased access to online content 69% 73% 73%

Learning is more personalized 65% 60% 57%

Students are collaborating more 57% 58% 58%

Improved teacher-student communications 48% 58% 63%

But what metrics are school and district leaders using to measure outcomes from mobile learning?

Turn to a neighbor and share: How are you evaluating the success of your mobile learning programs?

How satisfied are you with your evaluation process?

Audience feedback

Administrators say these are the top ways they are measuring the impact of their digital initiatives:

Metrics % of district administrators

nationwide

Which ones are you using?

Student engagement levels 77%

Classroom observations 69%

Student feedback 62%

Student achievement results 60%

Teacher buy-in 54%

Teacher feedback 53%

Parent feedback 37%

Two of the 8 essentials:

✓Purposeful planning for mobile device usage

✓Measuring project results with meaningful metrics

✓Purposeful planning for mobile device usage

▪ What are your goals?▪ What outcomes do

you want to see? ▪ Why is this important

for your school or district?

✓Measuring project results with meaningful metrics

▪ Aligning metrics with goals

▪ Leveraging appropriate tools

▪ Being open to what you don’t know yet

Evaluating mobile learning begins with project planning

Introduction to 3 mobile learning projects, the metrics they used/are using to evaluate program success & lessons learned that can inform your efforts:

❑ “Making Learning Mobile” – Chicago

❑ “STEAMing Ahead with Mobile Learning – San Diego

❑ “Classroom of the Future” – Turkey

Let’s get real!

“Making Learning Mobile” – Chicago

o Three year focused study on the impact of extended Internet access and mobile learning on student achievement

o Study group: 5th grade students and their teachers at Falconer Elementary School in Chicago, IL

o Students each assigned a Samsung Android tablet with 4G LTE enabled education broadband to use at school and at home

Partners: Kajeet, Qualcomm Wireless Reach, Project Tomorrow


Objectives of the project was to learn more about:

✓ How did the students and teachers use the tablets and extended Internet connectivity for learning?

✓ What benefits did they derive from those experiences?✓ What was the impact of the experiences on student learning and overall

academic achievement?

Evaluation metrics included:

• Feedback data from students and teachers – analysis of changes over 3 years • Classroom observations to evaluate depth of usage and impact of home access • Analysis of student achievement results


Key findings from the study:

1. Tablet became an indispensable learning tool

2. Tablets and extended connectivity enabled greater communications between students and teachers

3. Benefits to students included development of college/career ready skills and new learning behaviors

4. Teachers’ skills, motivation and readiness for tablet integration impacted effectiveness of usage

5. Purposeful integration within math = higher student achievement and growth


Insights from the evaluation of this project:

▪ Build your evaluation methodology and tools to be flexible and open to “surprises”

▪ Understand that the teacher adoption/assimilation/adaption process may be a three year adventure – be reasonable with project goals and the timeline for intended outcomes, and adjust your logical framework accordingly

▪ Reality alert: traditional data collection and analysis tools may not work for mobile learning

▪ With mobile learning, it is essential to collect and analyze both quantitative and qualitative data – feedback matters

▪ Keep eyes wide open for lessons learned – these are gold!

“STEAMing Ahead with Mobile Learning” – San Diego

Partners: San Diego Public Library, e3 Civic High School, Qualcomm Wireless Reach, Project Tomorrow

9th graders were provided with 4GL tablet that included an AR app to learn about the STEAM elements within the Central Library Dome design, engineering and construction



✓ How a contextually relevant learning experience for students could introduce students to the role of STEAM within construction and design

✓ The role of tablets and 4GL in increasing student engagement in learning✓ How to effectively increase student interest in STEAM career fields ✓ The ways in which this application could be used within other curricular fields

and with the general public


• Feedback data from students and teachers• Observation of the learning activity with “person on the street” interviews • Analysis of changes in students’ interest levels in STEAM career fields


Key findings from the study:

1. Students said use of AR content increased their engagement in learning

2. Benefits included having opportunity to work collaboratively with classmates on a real world activity; enjoyment level surprised the students!

3. Different learning modalities within the experience appealed to different students: videos, images, text, assessment

4. 40% of the were more interested in a STEAM career field as a result of this learning experience

5. Teachers’ excited about how to use AR in other curricular areas; library staff exploring ways to bring same content to general public


Insights from the evaluation of this project:

▪ A small project evaluation period demands tighter adherence to project goals. Project goals therefore must be clearly defined with broad buy-in from all stakeholders and partners.

▪ Critical to use the right tools to collect data. Example: to show change in attitudes, leverage pre and post data collection to “see” process of discovery and new ideas

▪ Reality alert: students are not familiar with all new technologies and thus, the awareness/tool learning process must be taken into account within the analysis of impact

▪ With a new technology such as AR, bring teachers into the evaluation development process upfront so that the analysis of the outcomes is based upon a realistic understanding of the environment

“Classroom of the Future” – Turkey

Partners: Turkish Education Association, Qualcomm Wireless Reach

▪ 7th grade students and their teachers from 7 schools have Qualcomm-enabled tablets with wireless connectivity to support learning in school and at home

▪ Initial focus was on English instruction classes but project has expanded to include other subject areas now

▪ Teachers were provided with specialized professional development to support this project


✓ How the inclusion of mobile devices in the learning environment enhances student learning and engagement particularly in reading and writing

✓ How the mobile learning experience improves students’ perceptions of their technology skills especially as an important global workplace skill

✓ If the teacher PD and the use of the devices in the classroom results in increases in teachers’ capacity, comfort and confidence in implementing mobile learning


• Feedback data from students and teachers – pre and post surveys • Aligning pre and post survey data elements directly with project goals • Linking teacher data with student data for analysis of impact

Key findings from the study (so far!):

1. Students and teachers agree: the use of the tablets with wireless connectivity both at school and at home are important to support learning.

2. 3/4 of the teachers reported: mobile learning environment resulted in improved student engagement, enhanced quality of their students’ work and students taking greater responsibility for their own learning.

3. The availability of the 1:1 tablets for student use at school and at home increased the students’ overall use of technology for learning - 86% of the students developed stronger technology skills during the school year.

4. With the new tools and training, 88% of the teachers reported greater communications with their students and 76% noted an increase in their overall effectiveness as a teacher.

Interim insights from the evaluation of this project:

▪ Even as project expands to include new teachers and schools, it is important to keep the objectives and evaluation plan stable so that you can evaluate for growth and maturity alongside the success of the replication.

▪ Analyzing teacher and student data at the class/school level side by side provided valuable insights into the relationship between usage, personal valuation and impact.

▪ Monitoring is not the same as evaluation. But monitoring is important for a valid evaluation.

▪ A good evaluation demands good data. To get good data, you have to ask the right questions.

Selection of tips and strategies you should think

about when evaluating your own mobile learning

projects

8

Selection of tips and strategies to use when evaluating your own mobile learning projects

1. Evaluation is not magic. Write out a plan and do that at the same time you are developing your operational plan for implementation so it aligns to your objectives for your project .



2. The quality of the data will determine the validity of your evaluation – and thus, what you will learn. You should always worry about the validity of your data!




3. Use the right tool for the right objectives in your plan. Some tools work better than others for certain objectives.




3. Use the right tool for the right objectives in your plan. Some tools work better than others for certain objectives.

4. Evaluation is ongoing – full year and year to year


5. Just because you use a mobile device and you are in education don’t assume that you know how to evaluate a mobile learning project. Evaluation is tricky business to get right!


5. Just because you use a mobile device and you are in education does not mean that you automatically know how to evaluate a mobile learning project. This is tricky business to get right!

6. Keep your eyes wide open for surprises and unintended outcomes –mobile learning is not “jello” yet. This field is still developing and there is no recipe or formula for success yet.


5. Just because you use a mobile device and you are in education does not mean that you automatically know how to evaluate a mobile learning project. This is tricky business to get right!

6. Keep your eyes wide open for surprises and unintended outcomes –mobile learning is not “jello” yet. This field is still developing and there is no recipe or formula for success yet.

7. Be realistic about your outcomes especially regarding the timeframes in which you expect to see those outcomes. To get to the big impact outcomes takes a lot of work in the classroom!


8. Most importantly, an evaluation of a mobile learning project is about so much

more than mobile devices. It is really about the transformative learning

experiences at your school.

Take advantage of that!

More resources available for extended learning

https://www.tomorrow.org/publications/MakingLearningMobile.html▪ 3 years of evaluation reports from the Making Learning Mobile project

https://www.qualcomm.com/media/documents/files/united-states-making-learning-mobile-mlm-project.pdf▪ Case study on the Qualcomm Wireless Reach site

Additional information about the projects featured today:

Making Learning Mobile – Chicago

https://www.tomorrow.org/publications/MakingLearningMobile.html

https://www.qualcomm.com/media/documents/files/united-states-making-learning-mobile-mlm-project.pdf

More resources available for extended learning

Additional information about the projects featured today:

STEAMing Ahead with Mobile Learning – San Diego

https://www.qualcomm.com/media/documents/files/united-states-steaming-ahead-with-mobile-learning.pdf

Classroom of the Future – Turkey

(not published yet – hard copies provided today for you!)

https://www.qualcomm.com/media/documents/files/united-states-steaming-ahead-with-mobile-learning.pdf

More resources available at www.tomorrow.org

National Speak Up reports and infographics

Targeted and thematic reportsDigital learning trendsCommunity engagement Mobile learning – infographic! Games in the classroomBlended learning outcomes

Presentations, podcasts and webinars

Services: consulting, workshops, evaluation and efficacy studies✓ We have expertise in the evaluation of mobile implementations, new

classroom models and digital content usage

New Speak Up 2017 reports released thru Sept 2018

Plan now for your district to participate in Speak Up 2018

Online surveys for: o K-12 students – individual + groupo Parents – English and Spanisho Teachers o Librarians/Media Specialistso School Site & District Administratorso Technology Leaderso Community Members &

Communications Officers

Surveys open Oct 2018 – Jan 2019Learn more www.tomorrow.org/speakup

100% free service to all schools and districts – including reports with comparison data

Your questions,

comments,

thoughts

Best Practices for Evaluating

Mobile Learning:

Real Stories from Around the Globe

Dr. Julie A. EvansCEO, Project Tomorrow June 25, 2018 @SpeakUpEd@JulieEvans_PT

Documents

Best Practices for Evaluating Mobile Learning: Real ... 2018 Mobile Learning Eval... · Benefits of mobile learning Elementary school principals Middle school principals High school