Evaluating Learning Systems

  • View
    215

  • Download
    0

Embed Size (px)

Text of Evaluating Learning Systems

  • 8/9/2019 Evaluating Learning Systems

    1/70

  • 8/9/2019 Evaluating Learning Systems

    2/70

    Six Facets of

    Instructional Product Evaluation

    Review

    Effectiveness

    Mainten

    ance

    Form

    ative

    Needs

    Assessment

    Impact

  • 8/9/2019 Evaluating Learning Systems

    3/70

    Development ActivitiDevelopment Activiti

    Product Conceptualization

    Design

    Development

    Project Re-conceptualization

    Implementation

    Institutionalization

    valuation Functionsvaluation Functions

    Review

    Needs Assessment

    Formative Evaluation

    Maintenance Evaluation

    Effectiveness Evaluation

    Impact Evaluation

  • 8/9/2019 Evaluating Learning Systems

    4/70

  • 8/9/2019 Evaluating Learning Systems

    5/70

    Dimensions of effective technology

    enhanced learning environments:

    q Task-Orientedq Challenging

    q Collaborativeq Constructionistq Conversationalq Responsiveq Reflectiveq

    Formative

  • 8/9/2019 Evaluating Learning Systems

    6/70

    Task-Oriented

    The tasks faculty set

    for students define theessence of the

    learning environment.

    If appropriate, tasksshould be authentic

    rather than academic.

    Academic Authentic

  • 8/9/2019 Evaluating Learning Systems

    7/70

    Task-Oriented Example

    Students in online

    instructional

    design courses

    are tasked with

    designing

    interactive

    modules for real

    clients.

  • 8/9/2019 Evaluating Learning Systems

    8/70

    Challenging

    The notion that

    interactive learning iseasy should be

    dispelled. Learning is

    difficult and studentsshould not spoon fed

    simplified versions of

    their fields of study.

    Simple Complex

  • 8/9/2019 Evaluating Learning Systems

    9/70

    Challenging Example

    In a Masters of

    Public Health

    program, studentsconfront problems

    as complex and

    difficult as the onestheyll face in the

    real world.

  • 8/9/2019 Evaluating Learning Systems

    10/70

    Collaborative

    Web-based tools for

    group work andcollaboration can

    prepare students

    for team work in21st Century

    work environments.

    Unsupported Integral

  • 8/9/2019 Evaluating Learning Systems

    11/70

    Collaborative Example

    Art, dance, andmusic students arecollaborating to

    produce onlineshows with digitalversions of theirworks andperformances forcritique byinternational experts.

  • 8/9/2019 Evaluating Learning Systems

    12/70

    Constructionist

    Faculty should

    engage students increating originalknowledge

    representationsthat can be shared,critiqued, andrevised.

    Replication Origination

  • 8/9/2019 Evaluating Learning Systems

    13/70

  • 8/9/2019 Evaluating Learning Systems

    14/70

    Conversational

    Students must have

    ample time andsecure spaces for

    in-depth

    discussions,debates, arguments,

    and other forms of

    conversation.

    One-way Multi-faceted

  • 8/9/2019 Evaluating Learning Systems

    15/70

    Conversational Example

    New knowledge and insight are beingconstructed in conversation spaces

    such as the e-learning forums found in

    BlackBoard, WebCT, Desire2Learn,and other

    online

    learning

    authoring

    platforms.

  • 8/9/2019 Evaluating Learning Systems

    16/70

  • 8/9/2019 Evaluating Learning Systems

    17/70

    Responsive Example

    This is an areawhere R&D areneeded. Some

    universities areseeking to establishsupportive onlinenetworks that willcontinue throughouta career, indeedthroughout a life.

  • 8/9/2019 Evaluating Learning Systems

    18/70

    Reflective

    Both faculty and

    learners mustengage in deep

    reflection and

    metacognition. Theseare not instinctive

    activities, but they

    can be learned.

    Shallow Deep

  • 8/9/2019 Evaluating Learning Systems

    19/70

    Reflective Example

    Teacher preparation

    students are keeping

    electronic journals toreflect upon the

    children they teach,

    and their roles asadvocates for

    children.

  • 8/9/2019 Evaluating Learning Systems

    20/70

    Formative

    Learningenvironments can bedesigned to allowstudents to developprototype solutions

    over time rather thanto find one rightanswer that someone

    else has defined.

    Fixed Assessment Developmental

  • 8/9/2019 Evaluating Learning Systems

    21/70

    Formative Example

    Faculty shouldengage their

    students in ongoing

    efforts to evaluateand refine their

    work related to

    authentic tasks toencourage lifelong

    learning.

  • 8/9/2019 Evaluating Learning Systems

    22/70

    Task-Oriented

    Challenging

    Collaborative

    Constructionist

    Conversational

    Responsive

    Reflective

    Formative

    Traditional Course Online Course

  • 8/9/2019 Evaluating Learning Systems

    23/70

    Heuristic Review

  • 8/9/2019 Evaluating Learning Systems

    24/70

    What is usability?

    The concern withdesigning softwareapplications which

    people find easy touse and personallyempowering.

    Usable computerprograms are logical,intuitive, and clear tothe people who use

    them.

  • 8/9/2019 Evaluating Learning Systems

    25/70

    Web Site Usability

    The most common user actionon a Web site is to flee.

    at least 90% of all commercial

    Web sites are overly difficult touse..the average outcome ofWeb usability studies is that testusers fail when they try to performa test task on the Web. Thus, whenyou try something new on theWeb, the expected outcome isfailure.

    Jakob Neilsen

    Edward Tufte

  • 8/9/2019 Evaluating Learning Systems

    26/70

    Typical Web Usability Problems

    bloated page design

    internally focused

    design

    obscure site

    structures

    lack of navigation

    support writing style

    optimized for print Jakob Neilsen

    http://www.useit.com

  • 8/9/2019 Evaluating Learning Systems

    27/70

    Key Usability Principles

    Structure - organizemeaningfully

    Simplicity - makecommon tasks easy

    Visibility - all dataneeded for a task

    Feedback - keep usersinformed

    Tolerance - allow cancel,back

    Reuse - reduce the

    users' need to remember

  • 8/9/2019 Evaluating Learning Systems

    28/70

    Nielsens Web Usability Rules Visibility of system status

    Match between system andreal world

    User control and freedom

    Consistency and standards

    Error prevention Recognition rather than recall

    Flexibility and efficiency ofuse

    Help users recognize,diagnose, and recover fromerrors

    Help and documentation

    Aesthetic and minimalistdesign

  • 8/9/2019 Evaluating Learning Systems

    29/70

    Two Major Ways to Evaluate Usability

    Heuristic Review

    quick and relatively inexpensive

    based on expert analyses

    no user involvement

    Usability Testing

    finds more problems

    user involvement increases validity

    when designers see problems live, it

    has a huge impact

  • 8/9/2019 Evaluating Learning Systems

    30/70

    Heuristic Review

    Several experts individually compare a

    product to a set of usability heuristics

    Typical heuristic:

    Visibility of

    system status The system should

    always keep users

    informed about

    what is going on,through appropriate

    feedback within

    reasonable time.

  • 8/9/2019 Evaluating Learning Systems

    31/70

    Heuristic Review

    Violations of the heuristics areevaluated for theirseverityand extent

    Severity Scale:1 Cosmetic: fix if possible.2 Minor: fixing this should be given low priority.3 Medium: fixing this should be given medium priority.4 Major: fixing this should be mandatory before the system is

    launched. If the problem cannot be fixed before launch, ensurethat the documentation clearly shows the user a workaround.

    5 Catastrophic: fixing this is mandatory; no workaround possible.

    Extensiveness Scale:1 Single case2 Several places3 Widespread

  • 8/9/2019 Evaluating Learning Systems

    32/70

    Heuristic Review

    At a group meeting, violation reports arecategorized and assigned

    Heuristics violated are

    identified

    Average severity and

    extensiveness ratings are

    compiled

    Opportunities for

    improvement are clarified

    Feasible solutions are

    recommended

  • 8/9/2019 Evaluating Learning Systems

    33/70

    Heuristic Review

    Example ofOpportunity For Improvement

    Opportunity 1 (4 reports. Avg. Severity=2.25, Avg. Extent=2.34,Heuristics Used: 1, 3)

    Consider providing more user feedback about where they are andwhat they should do next. Examples cited: No Page progress indicator No indication of how to start

    Suggestions: Provide a page-progress indicator, such as page 3 of 12 Put a Click a section below to start: on the first screen, as a TOC

    header

  • 8/9/2019 Evaluating Learning Systems

    34/70

    Heuristic Review

    Disadvantages

    Advantages Quick: Do not need to find or schedule users

    Easy to review problem areas many times

    Inexpensive: No fancy equipment needed

    Validity: No users involved