Download pdf - Evaluating Learning Systems

8/9/2019 Evaluating Learning Systems

1/70


2/70

Six Facets of

Instructional Product Evaluation

Review

Effectiveness

Mainten

ance

Form

ative

Needs

Assessment

Impact


3/70

Development ActivitiDevelopment Activiti

Product Conceptualization

Design

Development

Project Re-conceptualization

Implementation

Institutionalization

valuation Functionsvaluation Functions

Review

Needs Assessment

Formative Evaluation

Maintenance Evaluation

Effectiveness Evaluation

Impact Evaluation


4/70


5/70

Dimensions of effective technology

enhanced learning environments:

q Task-Orientedq Challenging

q Collaborativeq Constructionistq Conversationalq Responsiveq Reflectiveq

Formative


6/70

Task-Oriented

The tasks faculty set

for students define theessence of the

learning environment.

If appropriate, tasksshould be authentic

rather than academic.

Academic Authentic


7/70

Task-Oriented Example

Students in online

instructional

design courses

are tasked with

designing

interactive

modules for real

clients.


8/70

Challenging

The notion that

interactive learning iseasy should be

dispelled. Learning is

difficult and studentsshould not spoon fed

simplified versions of

their fields of study.

Simple Complex


9/70

Challenging Example

In a Masters of

Public Health

program, studentsconfront problems

as complex and

difficult as the onestheyll face in the

real world.


10/70

Collaborative

Web-based tools for

group work andcollaboration can

prepare students

for team work in21st Century

work environments.

Unsupported Integral


11/70

Collaborative Example

Art, dance, andmusic students arecollaborating to

produce onlineshows with digitalversions of theirworks andperformances forcritique byinternational experts.


12/70

Constructionist

Faculty should

engage students increating originalknowledge

representationsthat can be shared,critiqued, andrevised.

Replication Origination


13/70


14/70

Conversational

Students must have

ample time andsecure spaces for

in-depth

discussions,debates, arguments,

and other forms of

conversation.

One-way Multi-faceted


15/70

Conversational Example

New knowledge and insight are beingconstructed in conversation spaces

such as the e-learning forums found in

BlackBoard, WebCT, Desire2Learn,and other

online

learning

authoring

platforms.


16/70


17/70

Responsive Example

This is an areawhere R&D areneeded. Some

universities areseeking to establishsupportive onlinenetworks that willcontinue throughouta career, indeedthroughout a life.


18/70

Reflective

Both faculty and

learners mustengage in deep

reflection and

metacognition. Theseare not instinctive

activities, but they

can be learned.

Shallow Deep


19/70

Reflective Example

Teacher preparation

students are keeping

electronic journals toreflect upon the

children they teach,

and their roles asadvocates for

children.


20/70

Formative

Learningenvironments can bedesigned to allowstudents to developprototype solutions

over time rather thanto find one rightanswer that someone

else has defined.

Fixed Assessment Developmental


21/70

Formative Example

Faculty shouldengage their

students in ongoing

efforts to evaluateand refine their

work related to

authentic tasks toencourage lifelong

learning.


22/70

Task-Oriented

Challenging

Collaborative

Constructionist

Conversational

Responsive

Reflective

Formative

Traditional Course Online Course


23/70

Heuristic Review


24/70

What is usability?

The concern withdesigning softwareapplications which

people find easy touse and personallyempowering.

Usable computerprograms are logical,intuitive, and clear tothe people who use

them.


25/70

Web Site Usability

The most common user actionon a Web site is to flee.

at least 90% of all commercial

Web sites are overly difficult touse..the average outcome ofWeb usability studies is that testusers fail when they try to performa test task on the Web. Thus, whenyou try something new on theWeb, the expected outcome isfailure.

Jakob Neilsen

Edward Tufte


26/70

Typical Web Usability Problems

bloated page design

internally focused

design

obscure site

structures

lack of navigation

support writing style

optimized for print Jakob Neilsen

http://www.useit.com


27/70

Key Usability Principles

Structure - organizemeaningfully

Simplicity - makecommon tasks easy

Visibility - all dataneeded for a task

Feedback - keep usersinformed

Tolerance - allow cancel,back

Reuse - reduce the

users' need to remember


28/70

Nielsens Web Usability Rules Visibility of system status

Match between system andreal world

User control and freedom

Consistency and standards

Error prevention Recognition rather than recall

Flexibility and efficiency ofuse

Help users recognize,diagnose, and recover fromerrors

Help and documentation

Aesthetic and minimalistdesign


29/70

Two Major Ways to Evaluate Usability

Heuristic Review

quick and relatively inexpensive

based on expert analyses

no user involvement

Usability Testing

finds more problems

user involvement increases validity

when designers see problems live, it

has a huge impact


30/70

Heuristic Review

Several experts individually compare a

product to a set of usability heuristics

Typical heuristic:

Visibility of

system status The system should

always keep users

informed about

what is going on,through appropriate

feedback within

reasonable time.


31/70

Heuristic Review

Violations of the heuristics areevaluated for theirseverityand extent

Severity Scale:1 Cosmetic: fix if possible.2 Minor: fixing this should be given low priority.3 Medium: fixing this should be given medium priority.4 Major: fixing this should be mandatory before the system is

launched. If the problem cannot be fixed before launch, ensurethat the documentation clearly shows the user a workaround.

5 Catastrophic: fixing this is mandatory; no workaround possible.

Extensiveness Scale:1 Single case2 Several places3 Widespread


32/70

Heuristic Review

At a group meeting, violation reports arecategorized and assigned

Heuristics violated are

identified

Average severity and

extensiveness ratings are

compiled

Opportunities for

improvement are clarified

Feasible solutions are

recommended


33/70

Heuristic Review

Example ofOpportunity For Improvement

Opportunity 1 (4 reports. Avg. Severity=2.25, Avg. Extent=2.34,Heuristics Used: 1, 3)

Consider providing more user feedback about where they are andwhat they should do next. Examples cited: No Page progress indicator No indication of how to start

Suggestions: Provide a page-progress indicator, such as page 3 of 12 Put a Click a section below to start: on the first screen, as a TOC

header


34/70

Heuristic Review

Disadvantages

Advantages Quick: Do not need to find or schedule users

Easy to review problem areas many times

Inexpensive: No fancy equipment needed

Validity: No users involved

Finds fewer problems (50% less in some cases)

Getting good experts can be challenging

Building consensus with experts is sometimes

difficult


35/70

Another Weakness

Some people

believe that

heuristic evaluationis too subjective.

Human judges areprone to poor

judgment at times.


36/70

Usability Standardshttp://www.astd.org/ASTD/marketplace/ecc/ecc_home

ASTD offers certification of e-learning courses,including 8 usability standards:

-Navigation

-Orientation-Feedback cues

-Link cues

-Links labeled-Help

-Legibility

-Text quality


37/70


38/70

Heuristics for E-Learning Evaluation

2. Match between system

and the real world: The

e-learning programs

interface employs words,

phrases and concepts

familiar to the learner orappropriate to the content,

as opposed to system-

oriented terms. Wherever

possible, the e-learningprogram utilizes real-world

conventions that make

information appear in a

natural and logical order.


39/70


3. Error Recovery andExiting: The e-

learning program

allows the learner to

recover from inputmistakes and

provides a clearly

marked exit to leave

the program without

having to go through

an extended dialogue.


40/70


41/70


42/70


43/70


5. Error prevention: The e-learning program is

carefully designed to prevent common problemsfrom occurring in the first place.


44/70


6. Navigation support:

The e-learning program

makes objects, actions,

and options visible so

that the user does not

have to rememberinformation when

navigating from one part

of the program to

another. Instructions foruse of the program are

always visible or easily

retrievable.


45/70


7. Aesthetics:Screen displays do

not contain

information that is

irrelevant, andbells and whistles

are not gratuitously

added to the e-

learning program.


46/70


47/70


48/70


8. Help and

documentation: The e-

learning program

provides help and

documentation that isreadily accessible to the

user when necessary.

The help provides

specific concrete stepsfor the user to follow. All

documentation is written

clearly and succinctly.


49/70


50/70


10.Message

Design: The

e-learning

program

presentsinformation in

accord with

sound

principles ofinformation-

processing

theory.


51/70


52/70


53/70


11.Learning Design:

The interactions in

the e-learning

program have been

designed in accordwith sound principles

of learning theory.


54/70


12.Media Integration: The inclusion of media in the

e-learning program serves clear pedagogical

and/or motivational purposes.


55/70


13.Instructional

Assessment:

The e-learning

program provides

assessmentopportunities that

are aligned with

the program

objectives andcontent.


56/70


14.Resources:

The

e-learning

program

providesaccess to all

the resources

necessary to

supporteffective

learning.


57/70

Review

The purpose of review is toensure that the development

team is well-informed about

previous work done in the area

during the early stages ofproduct conceptualization.

Designers must avoid recreating

the wheel.


58/70

Review

The two primary methods usedare reviewing the related

literature and reviewing

competing products.

Regularly reviewing competingproducts is a great professional

development practice.

I can dobetter

than this!


59/70

Needs Assessment

The purpose of needsassessment is to identify the

critical needs that an instructional

product is supposed to meet.

Needs assessment providesessential information to guide the

design phase of the development

process.


60/70

Needs Assessment

The primary methods are: task analysis,

job analysis, and

learner analysis.

One of the most important results is

a list of specific goals and

objectives that learners will

accomplish with the new product.


61/70


62/70

Formative Evaluation Provided the results are used,

formative evaluation usuallyprovides the biggest payoff forevaluation activities.

Clients may be reluctant to

accept the results of formativeevaluation, especially as aprogram nears completion.


63/70


The purpose is to estimateshort-term effectiveness in

meeting objectives.

It is a necessary, but

insufficient, approach todetermining the outcomes

of interactive learning.


64/70


Evaluating implementation isas important as evaluating

outcomes.

If you dont understand how

instructional products were

actually implemented, you

cant interpret results.

A connection with

the server could not

be established?


65/70

Impact Evaluation

The purpose is to estimate thelong-term impact on

performance, both intended

and unintended.

It is extremely difficult

to evaluate the impact

of interactive learning

products.


66/70

Impact Evaluation

Evaluating impact isincreasingly critical because of

emphasis on the bottom line.

More and more clients expect

impact evaluation to include

return-on-investment (ROI)

approaches.


67/70


The purpose of maintenanceevaluation is to ensure the

viability of an interactive

product over time.

Maintenance is one of the

weakest links of web-based

learning environments.


68/70


Document analysis, interviews,observations, and automated data

collection are among the methods

used in maintenance evaluation.

Very few education and training

agencies engage in serious

maintenance evaluation.

Planning is the key to successful


69/70

Planning is the key to successful

instructional product evaluation.

Evaluation requires good

planning, careful

implementation, and

systematic follow-up.

A major challenge is

getting clients to identify

the decisions they face. Clear decisions drive the

rest of the planning.

He ristics for E Learning E al ation


70/70


15.Feedback:

Thee-learning

program

provides

feedback that

is contextual

and relevant

to theproblem or

task in which

the learner is

What the heck?