Transcript
Page 1: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact
Page 2: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Six Facets of Instructional Product Evaluation

Review

Effectiveness

Mai

nten

ance

Form

ativ

e

Needs Assessm

ent

Impact

Page 3: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Development ActivitiesDevelopment Activities

Product Conceptualization

Design

Development

Project Re-conceptualization

Implementation

Institutionalization

Evaluation FunctionsEvaluation Functions

Review

Needs Assessment

Formative Evaluation

Maintenance Evaluation

Effectiveness Evaluation

Impact Evaluation

Page 4: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

EMCC Design Document

• Urban Science Course Environment

Page 5: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Dimensions of effective technology enhanced learning environments:

Task-Oriented Challenging Collaborative Constructionist Conversational Responsive Reflective Formative

Page 6: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Task-Oriented

The tasks faculty set for students define the essence of the learning environment. If appropriate, tasks should be authentic rather than academic.

Academic Authentic

Page 7: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Task-Oriented Example

Students in online instructional design courses are tasked with designing interactive modules for real clients.

Page 8: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Challenging

The notion that interactive learning is easy should be dispelled. Learning is difficult and students should not spoon fed simplified versions of their fields of study.

Simple Complex

Page 9: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Challenging Example

In a Masters of Public Health program, students confront problems as complex and difficult as the ones they’ll face in the real world.

Page 10: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Collaborative

Web-based tools for group work and collaboration can prepare students for team work in21st Century work environments.

Unsupported Integral

Page 11: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Collaborative Example

Art, dance, and music students are collaborating to produce online shows with digital versions of their works and performances for critique by international experts.

Page 12: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Constructionist

Faculty should engage students in creating original knowledge representations that can be shared, critiqued, and revised.

Replication Origination

Page 13: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Constructionist Example

Students in fields ranging from aero-engineering to zoo management are producing digital portfolios as integral components of their academic programs.

Page 14: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Conversational

Students must have ample time and secure spaces for in-depth discussions, debates, arguments, and other forms of conversation.

One-way Multi-faceted

Page 15: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Conversational Example

New knowledge and insight are being constructed in conversation spaces such as the e-learning forums found in BlackBoard, WebCT, Desire2Learn,and other online learning authoring platforms.

Page 16: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Responsive

In learning communities, both faculty and students have a mutual responsibility to respond quickly, accurately, and with respect.

Superficial Genuine

Page 17: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Responsive Example

This is an area where R&D are needed. Some universities are seeking to establish supportive online networks that will continue throughout a career, indeed throughout a life.

Page 18: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Reflective

Both faculty and learners must engage in deep reflection and metacognition. These are not instinctive activities, but they can be learned.

Shallow Deep

Page 19: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Reflective Example

Teacher preparation students are keeping electronic journals to reflect upon the children they teach, and their roles as advocates for children.

Page 20: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Formative

Learning environments can be designed to allow students to develop prototype solutions over time rather than to find one right answer that someone else has defined.

Fixed Assessment Developmental

Page 21: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Formative Example

Faculty should engage their students in ongoing efforts to evaluate and refine their work related to authentic tasks to encourage lifelong learning.

Page 22: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Task-Oriented

Challenging

Collaborative

Constructionist

Conversational

Responsive

Reflective

Formative

Traditional Course Online Course

Page 23: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristic Review

Page 24: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

What is usability?

• The concern with designing software applications which people find easy to use and personally empowering.

• Usable computer programs are logical, intuitive, and clear to the people who use them.

Page 25: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Web Site Usability

• “ The most common user action on a Web site is to flee.”

• “at least 90% of all commercial

Web sites are overly difficult to use…..the average outcome of Web usability studies is that test users fail when they try to perform a test task on the Web. Thus, when you try something new on the Web, the expected outcome is failure.” Jakob Neilsen

Edward Tufte

Page 26: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Typical Web Usability Problems

• bloated page design• internally focused

design• obscure site

structures • lack of navigation

support • writing style

optimized for print Jakob Neilsen

http://www.useit.com/

Page 27: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Key Usability Principles

• Structure - organize meaningfully

• Simplicity - make common tasks easy

• Visibility - all data needed for a task

• Feedback - keep users informed

• Tolerance - allow cancel, back

• Reuse - reduce the users' need to remember

Page 28: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Nielsen’s Web Usability Rules• Visibility of system status • Match between system and

real world • User control and freedom • Consistency and standards • Error prevention • Recognition rather than recall• Flexibility and efficiency of

use • Help users recognize,

diagnose, and recover from errors

• Help and documentation • Aesthetic and minimalist

design

Page 29: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Two Major Ways to Evaluate Usability

• Heuristic Review– quick and relatively inexpensive– based on expert analyses– no user involvement

• Usability Testing– finds more problems– user involvement increases validity– when designers see problems live, it

has a huge impact

Page 30: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristic Review Several experts individually compare a

product to a set of usability heuristics

– Typical heuristic: • Visibility of

system status • The system should

always keep users informed about what is going on, through appropriate feedback within reasonable time.

Page 31: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristic Review Violations of the heuristics are

evaluated for their severity and extent

Severity Scale:1 Cosmetic: fix if possible.2 Minor: fixing this should be given low priority.3 Medium: fixing this should be given medium priority.4 Major: fixing this should be mandatory before the system is

launched. If the problem cannot be fixed before launch, ensure that the documentation clearly shows the user a workaround.

5 Catastrophic: fixing this is mandatory; no workaround possible.

Extensiveness Scale:1 Single case2 Several places3 Widespread

Page 32: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristic Review At a group meeting, violation reports are

categorized and assigned

Heuristics violated are identified

Average severity and extensiveness ratings are compiled

Opportunities for improvement are clarified

Feasible solutions are recommended

Page 33: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristic Review• Example of Opportunity For Improvement

Opportunity 1 (4 reports. Avg. Severity=2.25, Avg. Extent=2.34,Heuristics Used: 1, 3)

Consider providing more user feedback about where they are andwhat they should do next. Examples cited: No Page progress indicator No indication of how to start

Suggestions: Provide a page-progress indicator, such as “page 3 of 12” Put a “Click a section below to start:” on the first screen, as a TOC

header

Page 34: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristic Review

• Disadvantages

• Advantages– Quick: Do not need to find or schedule users– Easy to review problem areas many times– Inexpensive: No fancy equipment needed

– Validity: No users involved– Finds fewer problems (50% less in some cases)– Getting good experts can be challenging– Building consensus with experts is sometimes difficult

Page 35: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Another Weakness

• Some people believe that heuristic evaluation is too subjective.

• Human judges are prone to poor judgment at times.

Page 36: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Usability Standardshttp://www.astd.org/ASTD/marketplace/ecc/ecc_home

ASTD offers certification of e-learning courses, including 8 usability standards:

-Navigation-Orientation-Feedback cues-Link cues-Links labeled -Help-Legibility-Text quality

Page 37: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation

1. Visibility of system status: The e-learning program keeps the learner informed about what is happening, through appropriate feedback within reasonable time. •red for a problem

•yellow for a warning •green for OK

Page 38: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation

2. Match between system and the real world: The e-learning program’s interface employs words, phrases and concepts familiar to the learner or appropriate to the content, as opposed to system-oriented terms. Wherever possible, the e-learning program utilizes real-world conventions that make information appear in a natural and logical order.

Page 39: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation

3. Error Recovery and Exiting: The e-learning program allows the learner to recover from input mistakes and provides a clearly marked “exit” to leave the program without having to go through an extended dialogue.

Page 40: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact
Page 41: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact
Page 42: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation

4. Consistency and standards: When appropriate to the content and target audience, the e-learning program adheres to general software conventions and is consistent in its use of different words, situations, or actions.

Page 43: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation5. Error prevention: The e-learning program is

carefully designed to prevent common problems from occurring in the first place.

Page 44: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation

6. Navigation support: The e-learning program makes objects, actions, and options visible so that the user does not have to remember information when navigating from one part of the program to another. Instructions for use of the program are always visible or easily retrievable.

Page 45: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation

7. Aesthetics: Screen displays do not contain information that is irrelevant, and “bells and whistles” are not gratuitously added to the e-learning program.

Page 46: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact
Page 47: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact
Page 48: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation8. Help and

documentation: The e-learning program provides help and documentation that is readily accessible to the user when necessary. The help provides specific concrete steps for the user to follow. All documentation is written clearly and succinctly.

Page 49: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation9. Interactivity: The e-learning program provides

content-related interactions and tasks that support meaningful learning.

Page 50: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation10.Message

Design: The e-learning program presents information in accord with sound principles of information-processing theory.

Page 51: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact
Page 52: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact
Page 53: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation11.Learning Design:

The interactions in the e-learning program have been designed in accord with sound principles of learning theory.

Page 54: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation12.Media Integration: The inclusion of media in the

e-learning program serves clear pedagogical and/or motivational purposes.

Page 55: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation13.Instructional

Assessment: The e-learning program provides assessment opportunities that are aligned with the program objectives and content.

Page 56: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation14.Resources:

The e-learning program provides access to all the resources necessary to support effective learning.

Page 57: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Review

• The purpose of review is to ensure that the development team is well-informed about previous work done in the area during the early stages of product conceptualization.

• Designers must avoid recreating the wheel.

Page 58: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Review

• The two primary methods used are reviewing the related literature and reviewing competing products.

• Regularly reviewing competing products is a great professional development practice.

I can do better

than this!

Page 59: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Needs Assessment

• The purpose of needs assessment is to identify the critical needs that an instructional product is supposed to meet.

• Needs assessment provides essential information to guide the design phase of the development process.

Page 60: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Needs Assessment

• The primary methods are: – task analysis, – job analysis, and– learner analysis.

• One of the most important results is a list of specific goals and objectives that learners will accomplish with the new product.

Page 61: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Formative Evaluation• The purpose is to collect

information that can be used for making decisions about improving interactive learning products.

• Formative evaluation starts with the earliest stages of planning and continues through implementation.

Page 62: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Formative Evaluation• Provided the results are used,

formative evaluation usually provides the biggest payoff for evaluation activities.

• Clients may be reluctant to accept the results of formative evaluation, especially as a program nears completion.

Page 63: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Effectiveness Evaluation• The purpose is to estimate

short-term effectiveness in meeting objectives.

• It is a necessary, but insufficient, approach to determining the outcomes of interactive learning.

Page 64: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Effectiveness Evaluation• Evaluating implementation is

as important as evaluating outcomes.

• If you don’t understand how instructional products were actually implemented, you can’t interpret results.

A connection with the server could not

be established?

Page 65: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Impact Evaluation• The purpose is to estimate the

long-term impact on performance, both intended and unintended.

• It is extremely difficult to evaluate the impact of interactive learning products.

Page 66: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Impact Evaluation• Evaluating impact is

increasingly critical because of emphasis on the bottom line.

• More and more clients expect impact evaluation to include “return-on-investment” (ROI) approaches.

Page 67: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Maintenance Evaluation• The purpose of maintenance

evaluation is to ensure the viability of an interactive product over time.

• Maintenance is one of the weakest links of web-based learning environments.

Page 68: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Maintenance Evaluation• Document analysis, interviews,

observations, and automated data collection are among the methods used in maintenance evaluation.

• Very few education and training agencies engage in serious maintenance evaluation.

Page 69: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Planning is the key to successful instructional product evaluation.

• Evaluation requires good planning, careful implementation, and systematic follow-up.

• A major challenge is getting clients to identify the decisions they face.

• Clear decisions drive the rest of the planning.

Page 70: Six Facets of Instructional Product Evaluation Review Effectiveness Maintenance Formative Needs Assessment Impact

Heuristics for E-Learning Evaluation15.Feedback:

The e-learning program provides feedback that is contextual and relevant to the problem or task in which the learner is engaged.

What the heck?


Recommended