Embed Size (px)
Citation preview
Distribution Statement A Approved for public release: distribution unlimited.
Science of Learning and Readiness
(SoLaR) Recommendation Report
Science of Learning Practices for Distributed
Online Environments
30 April 2020
This work was supported by the U.S. Advanced Distributed Learning (ADL) Initiative (HQ0034-19-C-0015). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either
expressed or implied, of the ADL Initiative or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes.
Standard Form 298 (Rev. 8/98)
REPORT DOCUMENTATION PAGE
Prescribed by ANSI Std. Z39.18
Form Approved OMB No. 0704-0188
The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE 3. DATES COVERED (From - To)
4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER
5b. GRANT NUMBER
5c. PROGRAM ELEMENT NUMBER
5d. PROJECT NUMBER
5e. TASK NUMBER
5f. WORK UNIT NUMBER
6. AUTHOR(S)
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)
11. SPONSOR/MONITOR'S REPORT NUMBER(S)
12. DISTRIBUTION/AVAILABILITY STATEMENT
13. SUPPLEMENTARY NOTES
14. ABSTRACT
15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF: a. REPORT b. ABSTRACT c. THIS PAGE
17. LIMITATION OF ABSTRACT
18. NUMBER OF PAGES
19a. NAME OF RESPONSIBLE PERSON
19b. TELEPHONE NUMBER (Include area code)
Science of Learning and Readiness (SoLaR) Recommendation Report: SCIENCE OF LEARNING PRACTICES FOR DISTRIBUTED ONLINE ENVIRONMENTS
31UU
Recommendation Report
U
PE# 0603769D8Z
HQ0034-19-C-0015
U
Scotty D. Craig, Ph.D. Noah L. Schroeder
Nick Armendariz
OUSD Personnel & Readiness Advanced Distributed Learning Initiative 13501 Ingenuity Drive, Suite 248 Orlando, Florida 32826
407-384-5550
This report provides recommendations on best practices for supporting and creating modern learning ecosystems. The recommendations are based on findings from our two previous reports. The first report, the State of the Art Report, was a review of the state of the art for distributed learning environments that searched for best practices with evidence of supporting learning and improving learning organizations. The second report, The Exemplar Report, was a review of exemplar learning organizations that have successfully grown their learning enterprise.
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited
30/04/2020
OUSD/P&R/FE&T/ADLI
Arizona State University
U
Science of Learning and Readiness (SoLaR)
Recommendation Report:
SCIENCE OF LEARNING PRACTICES FOR DISTRIBUTED ONLINE
ENVIRONMENTS
HQ0034-19-C-0015
Submitted: 4/30/2020
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited OTHER REQUESTS FOR THIS DOCUMENT SHALL BE REFERRED TO
THE ADVANCED DISTRIBUTED LEARNING (ADL) INITIATIVE DIRECTOR.
2
Acknowledgements:
Principal Authors
Scotty D. Craig
Noah L. Schroeder
3
TABLE OF CONTENTS
Executive Summary ........................................................................................................................ 5
Recommendations for Learning Organizations .............................................................................. 8
Origins of the Recommendations ................................................................................................ 8
Summary of Best Practice Recommendation .............................................................................. 9
Institutional Recommendations ...................................................................................................... 9
Network of Education Expertise.............................................................................................. 9
Establish Human-Centered Evaluation Processes ................................................................. 10
Support Organizational Trust ................................................................................................ 11
Support Institution-Student Trust .......................................................................................... 12
Student Academic/Social Support Services .......................................................................... 12
General Support Infrastructure .............................................................................................. 14
General IT Infrastructure – Including Academic Courseware, Analytics Infrastructure ...... 15
Flexible Class Sizes ............................................................................................................... 15
Course Creation Recomendations ................................................................................................. 16
Course-level Recomendations ................................................................................................... 16
Data Supported Courses ........................................................................................................ 16
Support Self-Regulation ........................................................................................................ 17
Support Motivation ................................................................................................................ 18
Support Interaction Through Course Features....................................................................... 18
Foster a Community Of Inquiry ............................................................................................ 19
Content Design .......................................................................................................................... 21
Provide Scaffolding, Including Microlearning And Feedback .............................................. 21
Designed According To Human Cognitive Processes ........................................................... 22
Provide Video Content That Integrates Best Practices ................................................................. 22
End of Report Appendixes ............................................................................................................ 24
Appendix A ............................................................................................................................... 24
Appendix B ............................................................................................................................... 28
4
5
EXECUTIVE SUMMARY
The Science of Learning and Readiness (SoLaR) project seeks to demonstrate to Defense and
other Government stakeholders the “art of the possible” for high-quality distributed learning and
to create a practical guide for how to infuse such qualities into the broader Department of
Defense (DoD) distributed learning ecosystem. This report is the third in a series of three reports
examining the state of the art for advancing the applications of distributed learning. This report
provides recommendations for setting up modern learning ecosystems that incorporate
distributed learning techniques. The report consists of specific, actionable recommendations the
DoD can use for enhancing its distributed/blended learning institutions (i.e., the overall system),
courseware, and associated learning strategies/tactics.
This report provides recommendations on best practices for supporting and creating modern
learning ecosystems. The recommendations are based on findings from our two previous reports.
The first was a review of the state of the art for distributed learning environments that searched
for best practices with evidence of supporting learning and improving learning organizations.
The second report was a review of exemplar learning organizations that have successfully grown
their learning enterprise. The summarized findings from our review of best practices are
provided below as four high-level results:
The basic principles of human learning from the learning sciences still hold within
learning-at-scale environments.
Human learning within these environments must be supported by technology.
The technology must provide data on the learning process to the learning organization’s
stakeholders.
Learning organizations must use data to support: 1) the learners in the form of learning,
social, and academic guidance, and 2) the members of the learning institutions for
providing training, support, and recognition.
Our findings from interviews of exemplar learning organizations are summarized below:
A “think-before-you-act” mentality is needed for successfully growing the learning
initiative. Detailed planning and flexible policies should be in place before and as the
organization grows.
A technological infrastructure is needed that provides data on learner success and staff
training and performance.
Training is needed to keep all stakeholders on the same page.
Knowledgeable, responsive leadership and cross-sector strategies/policies are needed to
successfully grow.
Description of Report
6
The findings of the two previous reports have been condensed into a series of recommendations
at the institution and course creation level. At the institution level, we provide eight specific
recommendations. These recommendations include human-level strategies (e.g., educational
networks, human-centered evaluations, facilitating trust) and organization support-level
strategies (e.g., support services, humans and technological support infrastructures, and class size
flexibility). At the course level, we provide eight recommendations that deal with course
infrastructure (e.g., data support, facilitating self-regulation, supporting motivation, facilitating
interaction and facilitating engagement/creativity with Community of Inquiry ideas) and content
design (e.g., scaffolding, cognitive constraints, video integration). The recommendations are
supported by two heuristic checklists that experts (independent external experts or internal
members of the organization educational network) can use to evaluate the recommendations at
the organization and course levels.
7
8
RECOMMENDATION REPORT:
SCIENCE OF LEARNING PRACTICES FOR
DISTRIBUTED ONLINE ENVIRONMENTS
This report provides recommendations on the best practices for supporting distributed online
environments. This report provides 16 recommendations based on the science of learning
literature and lessons learned from exemplar organizations. It provides examples of these
recommendations. Finally, we provide two heuristic checklists that can be used to make
observations of learning organizations and courses for compliance with the recommendations.
RECOMMENDATIONS FOR LEARNING ORGANIZATIONS
Learning organizations are rapidly adapting how they provide education and training. These
changes are both technology-driven and practical to provide education and training to greater
numbers of learners at a rapid pace. Many of these learners are immersed in online learning
environments. For example, there are estimated to be 6,651,536 students enrolled in online
education courses at the postsecondary level. These 6.7 million students account for 33.7% of the
current student population and are part of a growth trend of online learning which has continued
for the last 13 years in the United States.
ORIGINS OF THE RECOMMENDATIONS
This report provides recommendations on best practices for supporting and creating modern
learning ecosystems. The recommendations are based on findings from our two previous reports.
The first was a review of the state of the art for distributed learning environments that searched
for best practices with evidence of supporting learning and improving learning organizations.
The second report was a review of exemplar learning organizations that have successfully grown
their learning enterprise. The summarized findings from our review of best practices are
provided below as four high-level results:
1) The basic principles of human learning from the learning sciences still hold within
learning at scale environments.
2) Human learning within these environments must be supported by technology.
3) The technology must provide data on the learning process to the learning organization.
4) Learning organizations must use data to support both learners with learning, social, and
academic guidance, and organization members by providing needed training, support,
and recognition.
9
Our findings from interviews with exemplar learning organizations are summarized below:
1) A “think-before-you-act” mentality is needed for successfully growing the learning
initiative. Detailed planning and flexible policies should be in place before and as the
organization grows.
2) A technological infrastructure is needed that provides data on learner success and staff
training and performance.
3) Training is needed to ensure all stakeholders share a common understanding.
4) Knowledgeable, responsive leadership and cross-sector strategies/policies are needed to
grow successfully.
SUMMARY OF BEST PRACTICE RECOMMENDATIONS
The findings of the two previous reports have been condensed into a series of recommendations
at the institution and course creation level. At the institution level, we provide eight specific
recommendations. These recommendations include human-level strategies (e.g., educational
networks, human-centered evaluations, facilitating trust) and organization support-level
strategies (e.g., support services, humans and technological support infrastructures, and class size
flexibility). At the course level, we provide eight recommendations that deal with course
infrastructure (e.g., data support, facilitating self-regulation, supporting motivation, facilitating
interaction and facilitating engagement/creativity with Community of Inquiry ideas) and content
design (e.g., scaffolding, cognitive constraints, video integration). The recommendations are
supported by two heuristic checklists that experts (independent external experts or internal
members of the organization educational network) can use to evaluate the recommendations at
the organization and course levels.
INSTITUTIONAL RECOMMENDATIONS
The following recommendations are a set of considerations needed for implemented a learning
organization that can grow (scale up) and still meet the needs of its stakeholders and constituents.
These recommendations could be implemented via company policy, an overall commitment to
learning, and by building appropriate support systems into the organizations structure.
NETWORK OF EDUCATION EXPERTISE
A basic understanding and commitment to human learning at all levels of the organization, from
administrators to SMEs/instructors, is crucial. This culture must be set from top-level
administrators and supported throughout the organization. It is useful for all decision makers to
have some level of understanding of good practices. Individuals should have detailed knowledge
of learning principles with the trust/authority to support implementation throughout the
institution’s network. This is not a single person; rather, it is a group of people spread throughout
the levels of the organization, including educational specialists serving as higher level directors,
10
learning engineers, instructional designers, and SME/instructors that are domain-based
educational researchers.
EXAMPLE
Examples of designing strategies for learning within an organization include:
• Establishing a distinct set of guidelines that outline how the learning ecosystem will be
structured and how the ecosystem will function between stakeholders.
• Identifying the leaders and contributors to the learning ecosystem and its maintenance.
• Allowing for learning meetings among the learning network.
• Promoting understanding of learning with focused learning meetings or specialized
training for all levels of management.
• Building a learning community or learning network within the organization. Identify their
expertise within the organization.
• Calculating the financial impact to the organization.
• Collecting and interpreting learning analytics to improve the learning ecosystem’s design
and functionality.
ESTABLISH HUMAN-CENTERED EVALUATION PROCESSES
Organizations must understand the strengths, weaknesses and needs of all stakeholders. Modern
learning ecosystems are large, complicated structures with diverse stakeholders. To serve the
entire organization and make informed decisions, it is essential to understand the needs of
distinct groups. Human-centered evaluations should be used to evaluate the functionality of
computer and technological systems and to collect data on how humans function within a
learning ecosystem.
11
Human-centered design and evaluation will allow
for a better understanding of the overall
performance of the learning organization. It
should have useful data-gathering methods for all
levels of deployment. Early evaluation (e.g., needs
assessments, contextual interviews) allows for
understanding the needs of the users to better set
requirements and design support into the system.
Mid-level evaluation of systems (e.g., usability
testing) allows for identification of problems that
users will have with the system before it is
implemented on a larger scale. Late evaluation
(e.g., experimental testing and mixed method
interviewing) of completed systems provides
evidence of effectiveness. This cycle can increase confidence and engagement from end users.
This could be applied new learning supports integrated into classes or to better understand the
current state of areas within the organization.
EXAMPLE
Human-centered evaluations within eLearning courses might take various forms. For instance, a
usability evaluation might enroll students in online class shells, and then record errors and
navigation behaviors as they locate materials and perform tasks. Such usability tests can span
observational methods (e.g., digital observation via screen capture software) or think-aloud
procedures (e.g., via video conferencing) wherein students talk about what they are attempting to
accomplish. Within a larger organization, survey methods can be used to develop an
understanding of general knowledge or perceptions about proposed implementations.
SUPPORT ORGANIZATIONAL TRUST
Organizations must also foster a sense of trust at all levels. In this case, “trust” broadly
encompasses confidence in and positive appraisals of available technologies and between
organization members. Administrators must encourage trust, foster relationships, and seek
common ground for discussion and action between stakeholders, while also collecting and using
data to facilitate change and support faculty in the online education endeavor. To be
“trustworthy,” administrative decision making should be guided by evidence-based tools and
metrics. Evaluation of stakeholder performance should be fair and transparent, and evaluations
should incorporate feedback from stakeholders. It should be grounded in policy that includes
recognition of stakeholders’ contributions (e.g., compensation or acknowledgement of time
commitment).
EXAMPLES
12
Examples of helping establish organizational trust include:
• Implementing clear evaluation metrics.
• Developing performance evaluations that follow metrics and are transparent.
• Providing clear and actionable performance feedback.
• Recognizing stakeholders’ contributions to success.
• Providing visible acknowledgement of effort.
• Identifying and providing support for online instructors through technical assistance and
course design/development assistance.
• Establishing clear workload and compensation models based on metrics and effort.
• Establishing course quality control methods to ensure accessibility and engagement.
SUPPORT INSTITUTION-STUDENT TRUST
Trust is crucial at the student level. Students’ trust in instructors and class content directly
impacts course grades. Student trust can be established by social engineering of the learning
environment (i.e., building logical class structures that minimize negative events), frequent
communication, maintaining a positive and consistent instructor persona, supporting peer-to-peer
mutual interaction (e.g., collaboration), involving students in decision making and
communication, defining clear policies, and creating clear and transparent oversight.
EXAMPLES
Examples of helping establish trust include:
• Eliminating student barriers to online learning by helping students locate and use external
resources, such as library services, tutoring services, and technology services.
• Minimizing anxiety over computer use and access through the provision of computer
access and training.
• Establishing positive rapport through respectful and fair treatment during online
interactions.
• Providing understandable and thorough instructions for student interactions with online
course content by providing an up-to-date syllabus, course goals, and realistic workload
expectations.
• Pre-emptively designing fair evaluation measures of course and student success based on
the nuances of online instruction.
• Explaining course grading and evaluations prior to beginning the course.
• Monitoring course completion and drop rates in real time to redeem struggling students.
STUDENT ACADEMIC/SOCIAL SUPPORT SERVICES
13
Learners must have adequate academic and social support structures from the institution. As
online distributed learning technologies continue to advance and propagate, the potential for
isolating students is a problem that must be addressed. Virtual interactions and asynchronous
environments may result in fewer opportunities for students to interact with peers or the
organization in meaningful ways. This can decrease the learner’s connection to the organization
and long-term engagement in the learning process.
Learning organizations must offer student support services and mindfully enable additional
social structures. There are several categories of support, such as academic services (e.g.,
advising, library, financial, and admissions) and social services (e.g., student organizations,
psychological services, placement services, and instructor support). These services interact with
and build upon other essential factors, including students’ family framework, personal
satisfaction, and perceived course relevance. All these elements play critical role in students’
decisions to persist or drop out of online courses.
EXAMPLES
Examples of student social support include:
• Providing students with
administrative contact
persons for individualized
counseling.
• Instituting institution-wide
communication standards
and evaluate
communication flow at
regular intervals.
• Making students aware of
financial obligations and
sources of help throughout
the duration of the course.
• Discussing opportunities
for inclusion in services
offered by institutional
services for persons with
disabilities.
• Making students aware of counseling services.
• Making library services available to all enrolled students.
14
GENERAL SUPPORT INFRASTRUCTURE
The learning organization needs a clear hierarchy of reporting, networks of support, and training
on technology and procedures. This support infrastructure will help to ease the burden of course
creation on instructors and allow them to focus on higher quality content. This support includes:
• Providing educational design support from instructional designers.
• Providing learning strategy support from learning scientists or discipline-based
educational researchers.
• Providing learning technology integration support on best practices for using technology
to support learning from learning engineers.
15
GENERAL IT INFRASTRUCTURE – INCLUDING ACADEMIC COURSEWARE,
ANALYTICS INFRASTRUCTURE
Technological infrastructure support should be at the core of any learning organization. Without
dedicated planning, organizations will struggle to deliver new technology= for users while also
(a) maintaining legacy systems beyond their reasonable lifespans, (b) seeking interoperability
between incompatible applications, and (c) doing so with dwindling resources. This requires a
variety of supporting infrastructures included appropriate policies and processes, information and
communication technologies, instructional support staff, technology hardware and facilities, and
training. This comprehensive support structure must be in place for instructors, staff, and
students with emphasis on technology support. Ideally, a modern infrastructure must transcend
vertical and isolated systems to embrace open data formats that can integrate data from across
the learning enterprise.
FLEXIBLE CLASS SIZES
Although learning at scale aims to provide worthwhile instruction to larger numbers of learners,
this goal does not mean that class sizes can grow infinitely. Appropriate class size is a
complicated question that must be considered by learning organizations, which should consider
(a) the type(s) of information being taught and (b) the technologies available to support the
learning environment. One generally recommended “rule” is that class size should be guided by
nature of the content (see Bloom’s taxonomy). Topics that require higher-level thinking (e.g.,
synthesis and evaluation) may be best suited to smaller class, whereas topics that entail lower-
level thinking (e.g., recall) may be taught in larger classes.
EXAMPLES
Examples of implementation of class size recommendations include:
• Planning for smaller class sizes when teaching high-level
content.
• Utilizing larger classes when foundational principles are the
class focus.
• Limiting class sizes, regardless of content, when learner or
instructor support is limited.
• Scaling courses at a commiserate pace with student
engagement ability.
• Planning for transition time and course re-development of
currently face-to-face courses.
• Optimizing technology for maintaining student interaction.
As material complexity increases class size should
decrease.
16
COURSE CREATION RECOMMENDATIONS
Course creation is going to be one of the most important expenses within learning organizations.
The adage “content is king” is popular for a reason. Without well-crafted and supported content
any learning enterprise will fail. To continue with the analogy, context is the kingdom. A king is
useless without a kingdom. The course should consist of well-structured content, but also must
have learning supports around it to create a robust course. How to build this support can vary
based on resources, culture, need, and implementation time. However, the recommendations
below provide a structure of required elements that can be implemented in some form regardless
of the individual composition of the learning organization.
COURSE-LEVEL RECOMMENDATIONS
DATA-SUPPORTED COURSES
Technology should collect and support data within courses. To modernize courses and enable
information sharing, learning technologies must be able to collect and output learning data.
Several data standards are already in use. For example, xAPI is a popular method for capturing,
standardizing, and sharing human performance data.
EXAMPLE
Use a standardized approach when capturing the interaction of students in the system (e.g.,
xAPI). An adapter can be developed to generate xAPI statements from existing or proprietary
systems. Utilize existing xAPI Profiles Specification to enable successful and semantically
interoperable xAPI implementation. Visualizations that use user data must be transparent to the
users (i.e., users can understand how the data led to the visualization). When designing a
visualization, relevant information to its intended user must be considered. The visualization
should incorporate interactivity to allow for visual exploration and to prevent overwhelming the
17
users. Information Seeking Mantra can serve as a guide when designing interactive
visualizations: overview first, zoom and filter, then details-on-demand.
SUPPORT SELF-REGULATION
Classes must support a learner’s self-regulation. Self-regulation is a broad term that refers to
one’s ability to monitor and regulate their learning experiences. Self-regulation skills are
essential for success in distributed learning environments. To successfully implement self-
regulation, the follow steps must be supported.
• Phase 1: Learners should be allowed to plan and
analyzing tasks for the class.
• Phase 2: Learners must be able to perform tasks and
enact their chosen strategies.
• Phase 3: Learners must be able to monitor their
performance and learning.
• Phase 4: Learners must be able to adapt future
learning efforts based on their observed results.
It should be noted that these steps are not always easy in
online settings. They can be facilitated by learning supports
such as visual dashboards. During this process students often need to be monitored by faculty
and to receive feedback, especially in Phase 2 when they are performing tasks.
EXAMPLES
A variety of ways to promote self-regulation exist in distributed learning contexts. One example
is providing learners with goal-setting templates to support learners' free choice in their learning.
It is a powerful method to help students unfold and frame their intrinsic goals or needs of
learning (e.g., personal growth, meaningful relationships with others, or contributing to the
community) instead of motivating them by extrinsic goals (e.g., fame, financial success, and
physical appearance) or outside pressures (e.g., social comparison with peers).
Examples of self-regulation support include:
• Student-facing learning analytic dashboards, which can be effective tools for developing
self-regulated learners because of their autonomous and informative feature (e.g., an
adaptive recommendation system and interactive data visualization, etc.).
• A recommendation system that aims for a more coarse-grained unit of learning (e.g.,
course modules) while allowing learners to explore additional materials.
• An intelligent tutoring system that aims for the fine-grained solutions to the learning
problems by providing specific feedback to learners as they work through the material.
18
SUPPORT MOTIVATION
Students exhibit varied levels of motivation while learning content in distributed as well as face-
to-face or blended learning environments. There are many theories of academic motivation that
examine motivation from different perspectives. Some of these theories suggest the need to
consider both internal and external motivational factors, such as an internal desire to succeed
versus the need for an external motivator. Knowing that students approach distributed learning
with varied levels and types of motivation can improve course designs by considering different
strategies for supporting students’ motivation to learn the content.
EXAMPLES
Foster student motivation can take on many forms. However, supporting motivation may differ
due to various types of content, context, and learner populations. For example, an individual
taking a course that is critical to their career may feel an internal desire to succeed, whereas one
taking a mandated course that has little relevance to their career may need more external
motivators to succeed.
One example of providing motivation is through the types of feedback and communication
provided to learners. For example, instilling a growth mindset, which is the perspective that one
can always improve their intelligence, can be beneficial. Praising effort and progress can provide
benefits to learners.
Personalized learning analytics applications such as student-facing learning analytics dashboards
(SFLADs) can be helpful to increase learners’ motivation by affording them ownership over
their own learning. These SFLADs can provide information such as their own learning
progression, trajectory, and goals so they can make decisions on whether they want to continue a
class or how they want to continue it. In short, SFLADs should empower students to become
data-informed learners. Specifically, a well-design SFLAD should focus on designing the
learning experience for learners which underscores the learners’ affective and non-cognitive
skills (e.g., interpersonal skills, persistence, communication skills, and other "soft" skills). In
addition, a variety of user experience design techniques (e.g., task analysis, questionnaires,
interviews, usability testing, etc.) should be utilized to examine and evaluate the effectiveness of
the SFLADs.
SUPPORT INTERACTION THROUGH COURSE FEATURES
Courses should be designed to support interactions. Building interactions with learning content is
a staple task for many instructional designers. The goal of instruction is often for students to
understand what the content means in the context of their own lives or careers, and therefore
having students purposefully interact with said content can be effective for facilitating learning.
This notion extends to distributed learning environments as well as face-to-face and blended
learning environments. Student interaction with course content can come in many forms,
19
including augmented or virtual reality experiences or simulations (AR and VR, respectively),
social media use, or cooperative learning opportunities. The appropriate use of each of these
technologies or techniques will depend on the content and course being taught.
There are also additional considerations regarding the use of social media and cooperative
learning activities, which include, but are not limited to:
• Making students aware of the risks and benefits of social media use in informal learning
opportunities.
• Training students in digital citizenship.
• Protecting students’ anonymity, privacy, and choice when using social media sites for
learning (e.g., FERPA considerations).
• Accommodating assignments for students who are unwilling to engage in social media
for learning.
• Instructing students on the location and platform for each assignment (LMS or social
media sites) so that there is no division of the location of learning materials.
EXAMPLE
Interactions with course content should be intentional and contextually relevant. Consider for
example, AR or VR learning environments. AR or VR experiences or simulations have been
used in a variety of fields. However, they have the potential to be time and cost intensive to
develop, and therefore may not be appropriate for every use case. There is also a broad range of
what these experiences or simulations may look like. For instance, a simple AR simulation may
entail additional information showing on a screen when the camera picks up an object within its
view, whereas a complex VR simulation could be used for training pilots.
FOSTER A COMMUNITY OF INQUIRY
Build interactions into the course. These interactions should include interacting with the content,
but also with the instructor and other students. In the past, distributed courses were critiqued for
a lack of interaction that made learners feel disconnected from the topic and class. Building
interactions with course content helps this by building a community of inquiry within the course.
20
A community of inquiry is a connected group of learners with common goals. It fosters critical
thinking, reflection, and discourse. To create this collaborative community, a class must have a
feeling of presence (being present) associated with it. There are thee type: cognitive, teaching,
and social.
• Cognitive presence is the ability of the learner to manipulate the content in relation to
their own lives.
• Teaching presence refers to how well the instructor designs and facilitates the course. The
instructor should be part of the course both visually in the form of video and as an
interactive partner by providing feedback and responses to student questions.
• Social presence refers to how cohesive the class of students is, or how well they openly
communicate. This can be facilitated by providing communication outlets for students
such as forums, discussions, or interactions on social systems such as Slack. However,
these interactions should be monitored by the instructor to identify incorrect information
or abuse.
EXAMPLES
Examples for increasing presence include:
• Establishing a strong teaching presence as a moderator of the social and cognitive
presences through using interaction opportunities and through establishing class
administration early in the course cycle.
• Helping students engage and collaborate through thoughtful, relevant, authentic group
learning activities.
• Segmenting content into sizes that suit the level of the learner (novice versus expert).
• Providing strong evaluation rubrics and feedback that reward students for effort in
multiple dimensions of class projects.
• Helping students connect with peers through intentional community engagement
activities.
• Helping students transition from traditional lecture-based learning to collaborative
learning through metacognitive strategies such as reflection.
• Ensuring students’ peer-peer and peer-instructor interactions are safe and respectful.
21
CONTENT DESIGN
PROVIDE SCAFFOLDING, INCLUDING MICROLEARNING AND FEEDBACK
Provide guidance (scaffolding) within eLearning courses. There are many types of scaffolding,
guidance, and feedback that can be critical elements of effective instruction in these contexts.
Microlearning techniques can help in this process. Within microlearning, larger content is
subdivided into smaller coherent topics. These topics can be provided to learners either in an
ordered sequence or to give extra guidance if learners are struggling with a specific concept.
EXAMPLES
Examples of providing scaffolding and guidance include, but are not limited to:
• Using authentic scenarios or cases that emphasize the general rules and principles of a
discipline and highlighting how these rules and principles can be transferred into new
situations.
• Providing worked examples to lower extraneous cognitive load for novice learners.
• Devising mechanisms for calling attention to unfamiliar vocabulary and principles or
systems, such as flashcards, low-stakes quizzing, and demonstrations.
• Setting realistic expectations for PBL/IL by remembering these tools promote meaning
and understanding and are not a means for knowledge acquisition.
• Providing human support and materials support, such as task overviews, scope of
learning overviews (learning objectives or goal statements), task requirements, and ideal
task representations so that students can emulate the ideal solution.
• Providing microlearning opportunities to provide just-in-time teaching and support.
Examples of providing feedback include, but are not limited to:
• Delivering supportive feedback through prompts that provide opportunities for instructors
to address domain-specific knowledge, guidance, and learner-strategies through
explanations, references, visual aids, and project templates.
• Encouraging reflective feedback using process maps, models of expert solutions, and
hints and questions to guide learners through the process and to identify knowledge gaps.
• Providing guidance to students by modeling good performance, providing task
explanations, setting accomplishable task expectations, and providing expert guidance.
• Delivering “feed-back” through giving specific, individual, timely, regular, detailed,
purposeful, task-appropriate, assignment-directed, and future-driven statements that are
expressed in concrete and understandable ways for all student assignments.
• Using peer feedback for the benefits of providing students with other perspectives on
their assignments.
22
DESIGNED ACCORDING TO HUMAN COGNITIVE PROCESSES
The limitations of human cognitive processing should guide the design of instruction and
instructional content. Research has shown that the working memory capacity is limited,
instructional sequences should be designed to prevent overloading the working memory capacity.
In addition to instructional sequencing, one must also consider the visual design of the materials
presented.
General recommendations from the research principles include:
• Avoiding presenting complex text and visual images unless there is adequate time to
process the information and guidance to facilitate understanding.
• Avoiding the use of images unless they are needed and relevant to the explicit point being
made. Images need to help the learner visualize the information. Otherwise, the images
will hinder learning.
• Presenting an exact narration of printed text should be avoided unless it is needed for a
specific learner population (For example visually impaired learners).
• Integrating narrative and visual information when presented at the same time.
• Presenting a complex diagram or complex written document by highlighting key points
when they are relevant. This might require taking a longer text or narrative information
and turning it into a series of subtopics with relevant linking information highlighted in
each.
EXAMPLE
A considerable number of instructional design principles can be used to help prevent overloading
the working memory capacity and improve learning. Examples include the modality principle,
spatial contiguity principle, temporal contiguity principle, worked example effect, signaling
principle, and more.
PROVIDE VIDEO CONTENT THAT INTEGRATES BEST PRACTICES
Courses should use instructional videos that follow specific research-based best practice
strategies. Video should not be just a long recording of a lecture as a replacement for a face-to-
face lecture. Ideally these videos should be short (five to 10 minutes at most) and should model
best practices, provide procedural knowledge (how-to knowledge), or brief, concise explanations
of concepts. When producing these videos, follow best practices for production, including
cognitive-based organizational principles such as using multiple modalities or signaling (visually
highlighting) important concepts. Create notes of the videos, follow the notes, and provide a
version of the notes to learners.
23
Video environment and instructor persona are also important. Stay professional but show your
personality. Similarly, make sure that the environment around the instructor is professional yet
relatable.
EXAMPLE
Examples of how to design effective
instructional videos include, but are not
limited to:
• Ensuring that videos follow
research-based instructional
design principles based on
human cognitive processes.
• Establishing that video content is
relevant to the topic and will
increase student engagement and
active learning.
• Utilizing video messages with
signaling and segmenting for
learners to identify the important
material quickly.
• Using videos that adhere to ADA accessibility guidelines.
• Considering the use of questions and reflection in videos for students to practice retrieval.
24
END OF REPORT APPENDIXES
APPENDIX A
Heuristic Evaluation Form – Organization Level
Instructions
This form is designed to help an organization determine how well it is
meeting the needs of its distributed learning constituents at the institutional level.
The person(s) completing this form should possess relevant expertise, such as a
learning scientist or learning engineering. This form is not intended to be all-
inclusive, but rather is designed to highlight the high-level requirements of an
effective distributed learning organization.
The organization can be categorized as one of four different ratings for each
criterion, which are defined as follows:
Rating Description
Exemplary The organization provides ample resources compared
to the current and projected needs. There are limited
areas for improvement.
Acceptable The organization provides the resources needed to
meet current needs. However, there are still areas of
potential improvement.
Adequate The organization provides the minimum resources to
meet current needs. More resources should be devoted
to the criterion. This rating is considered the minimum
needed for operation and may not be appropriate for
sustained operations.
Lacking The organization does not meet its constituents’ needs
and is not serving its constituents well.
25
Organizations should strive to reach Acceptable or, if possible, Exemplary for all criteria. An
organization with these ratings is likely in a good position to host effective distributed learning
courses. Note that meeting these criteria alone does not indicate an effective distributed learning
organization; rather, these criteria are indicative of having the capacity to be an effective
distributed learning organization.
An organization that currently provides distributed learning may find that they have criterion
rated at Adequate. As noted, this indicates a minimum level of proficiency and the organization
should conduct a needs analysis to find out the most appropriate way to improve in this criterion.
Any criterion that is rated as Lacking should perform a needs analysis for that criterion to better
understand why it is deficient, and then address these deficiencies before adding additional stress
onto the organization.
26
Organization-Level Criterion Summary
Topic Explanation
Network of Education
Expertise
Is there an organized structure of individual with educational
expertise throughout the organization. Are they structured so
they can support each other? Are they structured to support
strong educational practices throughout the organization?
Established human-centered
evaluation process
Does the institution have processes in place for formal or
informal usability evaluations of courses or learning
materials?
Support institutional trust Are there efforts, policies, and procedures in place to support
and understand trust within the organization?
Support institution-student
trust
Are there efforts, policies, and procedures in place to support
institution-student trust? (e.g., building logical class
structures, frequent communication, involving students in
decision-making, defining clear policies, etc.)
Student academic/social
support services
Are there ample student support services in place (e.g.,
providing students with administrative contact persons for
individualized counsel, instituting institution-wide
communication standards and evaluating communication
flow at regular intervals, making students aware of financial
obligations and sources of help, opportunities for inclusion in
services offered by institutional services for persons with
disabilities, etc.)?
General support infrastructure Are clear policies and supports in place so instructors know
who to approach or what to do if they encounter issues with
their courses?
General IT infrastructure Is the general IT infrastructure available to support online
courses at the scale the organization operates? For example,
is there appropriate capacity to support the number of
students accessing the system at once? Is an adequate
analytics infrastructure in place?
Allow for flexible class sizes Class size should be guided by the nature of the content and
the technology infrastructure available to support the course.
27
Learning Organization Heuristic Evaluation Form
Use the form below to review the policies and procedures for the learning organization. Rate
each category from 1 (Lacking) to 4 (Exemplary) based on the observed current state of the
organization. In the following comment box, provide explanation of your rating and feedback for
improvement.
Category Rating (1-4) Comments
Network of
Education
Expertise
Established
human-centered
evaluation
process
Support
administrator-
instructor trust
Support
institution-
student trust
Student
academic/social
support services
General support
infrastructure
General IT
infrastructure
Allow for
flexible class
sizes
28
APPENDIX B
Heuristic Evaluation Form – Course Level
Instructions
This form is designed to inform on the potential for a course to meet learner needs. The person(s)
completing this form should possess relevant expertise in learning science and learning
engineering principles. This form is not intended to be all inclusive, but rather is designed to
highlight the high-level requirements of an effective distributed learning course. In addition to
the criteria listed here, a course should also adhere to standard best practices of accessibility and
course design practices on assessment such as a clear link between content and assessment and
clear explanations of assessment criteria. This form is not intended to be used as an instructor
evaluation tool.
The course can be categorized as one of four different ratings for each criterion, which are
defined as follows:
Rating Description
Exemplary The course provides ample support for learners. There
are limited areas for improvement.
Acceptable The course provides the support needed to meet the
needs of learners. However, there are still areas of
potential improvement.
Adequate The course provides the minimum support to meet
learners needs. More support should be devoted to the
criterion. This rating is considered the minimum
needed for course delivery and may not be at
appropriate levels for all learners.
Lacking The course does not meet learners’ needs on this
criterion in the current state.
29
Ideally, courses will have ratings of Acceptable or Exemplary for all criterion. A course with
these ratings is likely in a good position to facilitate effective distributed learning. Note that
meeting these criteria alone does not indicate an effective distributed learning course. These
criteria are indicative of having the capacity to be an effective distributed learning course.
Effective course facilitation is essential in the learning process and largely unaccounted for by
this evaluation form.
A course may have criterion rated Adequate. This indicates a minimum level of proficiency and
the course designer should conduct a needs analysis to find out the most appropriate way to
improve in this criterion.
Any criterion that is rated as Lacking should perform a needs analysis of that criterion to better
understand why it is deficient before the course is provided to learners.
Course-Level Criterion Summary
Course Structure and Alignment
Topic Explanation
Fundamental data collection for analytics Does the course allow for the
collection of fundamental data that
could be used for analytics?
Examples (non-inclusive list) may
include quiz scores, final grades,
attendance, etc.
Support self-regulation Does the course include design
features that support student self-
regulation? Examples may include
features such as a calendar of
assignments and due dates, or an
analytics dashboard.
Support motivation Does the course include design
features to support student
motivation? Potential examples
include tying the content to their
career field or personal interests.
Support interaction with course features Does the course contain adequate
features to support student
interaction with the content (e.g.,
virtual reality, augmented reality,
social media, simulations, etc.)?
30
Fosters a Community of Inquiry Does the course support cognitive,
social, and teaching presence?
Content Design
Provides scaffolding, including microlearning and
feedback
Does the course include appropriate
scaffolding (e.g., extra readings,
videos, just-in-time teaching, or
microlearning opportunities)?
Does the course include frequent
opportunities for the student to
receive formative feedback?
Designed according to human cognitive processes There are a considerable number of
instructional design principles that
can be used to help prevent
overloading the working memory
capacity and improve learning (e.g.,
the modality principle, spatial
contiguity principle, temporal
contiguity principle, worked example
effect, signaling principle, etc.).
Provides video content integrates best practices Videos should follow research-based
instructional design principles based
on human cognitive processes. In
addition, videos can include content
that will increase student
engagement and active learning,
adhere to ADA accessibility
guidelines, and use questions and
reflection for students to practice
retrieval.
31
Course - Heuristic Learning Science Evaluation Form
Use the form below to review the policies and procedures for the learning organization. Rate
each category from 1 (Lacking) to 4 (Exemplary) based on the observed current state of the
organization. In the following comment box, provide explanation of your rating and feedback for
improvement.
Category Rating (1-4) Comments
Course Design
Fundamental
data collection
for analytics
Support self-
regulation
Support
motivation
Support
interaction with
course features
Fosters a
Community of
Inquiry
Content Design
Provides
scaffolding,
including
microlearning
and feedback
Designed
according to
human
cognitive
processes
Provides video
content
integrates best
practices
