Evaluating the Effectiveness of Instructional Design

in Online Offerings:

An Analytic Approach

Michael Cenkner, M.Ed.

Athabasca University

Agenda for this session:

Presenting a frameworkQuick overview of the framework

An example

Applying the framework Analyzing some examples together

ConclusionsUsefulness?

Limitations?

My professional context Digital resources - learnAlberta.caOnline courses - Working with facultyWeb. 2.0, etc. –ID within flood of new technologies

When something seems inadequate, how to say “why”?

A need a way to evaluate that is:Objective Shareable Analytical approach

An analytic framework: reverse-engineering the learning experience

ID analysis:Look at implementation

Identify model (inc.

learning obj’s)

Evaluate whether needs

may be met

ID development:Look at needs

Develop model (inc.

learning obj’s)

Implementation

Framework components (implicit or explicit) (Implicit or explicit)Objective• What should the learner be able to do as a result of this experience?Task• What is the activity they are doing in order to learn?Condition• What is going on around and within learners (before and during)?Physical Interaction• What do they physically do to learn this?Feedback• What thing happens in the environment as a result of the learner’s

actions?Criteria• How is the learning being measured?

Notes on the analysis

Components are implicit or explicit

• Explicit – They are identified (e.g. objectives)

• Implicit – You will need to identify

Looking for alignment,

not mere inclusion!

Guiding questions for componentsObjective• Are there one or more clear (explicit or implicit) objectives? • If there is an objective, is it appropriate for the given learners? Task• Is there a stated or implied task? Does the task have a beginning and an end? • Is the task an example of the kind of learning indicated in the objective? Condition• For the given task, do learners: Have motivation? • Prior knowledge and/or skills? • Adequate time? • Necessary resources? • Adequate orientation? Physical Interaction• Is the intended physical interaction (in all its steps or variations) clear? • Are learners physically able to engage as intended and necessary? • Are affordances available and useable as intended? Feedback• Is there feedback? From what source(s)? • Assumed? Necessary? Happening?• Timely? • An obviously direct result of the learner’s actions? Criteria• Are criteria provided? If so, are they summative only or formative as well? • If so, will the resulting framework appear reasonable, fair and motivating to the learners? • Do criteria address the same knowledge and skills as those intended to be developed through the intervention?

Objectives

“Evaluate the challenges of research in [this field] of management”

Tasks• Facilitate discussion on weekly (assigned) topic• Participate in discussion

Conditions

Weekly topic reflects topics related to research challenges in the field

Cohort group – have done one online course previously

Physical interaction

Participants read, write and post in Moodle discussion module.

Feedback

Student facilitator is participating at least twice.

Instructor gives weekly summary of each discussion.

Criteria

Quantity and quality of posts

Example 1, Objective 1

Objectives

“Demonstrate collaborative skills appropriate to … research in [this] field”

Tasks• Facilitate discussion on weekly (assigned) topic• Participate in discussion

Conditions

Participants are current practitioners doing a graduate certificate.

Each student doing own research proposal assignment

Physical interaction

Participants read, write and post in Moodle discussion module.

Feedback

Student facilitator is participating at least twice.

Instructor gives weekly summary of each discussion.

Criteria

Quantity and quality of posts

Example 1, Objective 2

Objectives

“Introduce students to the field of American literature as it has emerged and evolved in North American academic settings.”

Tasks

1. Read texts

2. Post answer to weekly assigned question

3. Participate in discussion; comment on one other student.

Conditions

Second-year undergraduate students

Discussion is not facilitated except informally by instructor

Physical interaction

Participants read, write and post in Moodle discussion module.

Feedback

Informally from other students and/or instructor

Criteria

Quantity and quality of posts

Example 2, Objective 1

Objectives

“Discuss the field of American literature in terms of the present globalizing moment that has produced challenges to the concepts of the nation-state.”

Tasks

1. Read texts

2. Post answer to weekly assigned question

3. Participate in discussion; comment on one other student.

Conditions

Second-year undergraduate students

Discussion is not facilitated except informally by instructor

Physical interaction

Participants read, write and post in Moodle discussion module.

Feedback

Informally from other students and/or instructor

Criteria

Quantity and quality of posts

Example 2, Objective 2

Example 3: Open Course-Ware from Carnegie-Mellon

Objectives

To greet someone in French

Tasks

1. Watch and listen to the dialogues

2. Identify which sentence is being said by listening

3. Order the sentences by dragging the words on-screen

Conditions

Learners are motivated.

Learners can replay dialogues at will.

Physical interaction

Use the mouse to control the play bar and links

Feedback

System indicates correct or incorrect answer.

Criteria

Sentence has been properly identified.

Sentence has been properly ordered

Example 3, Objective 1

Objectives

To pronounce the alphabet and basic sounds of French

Task

1. Listen to the sounds both separately and within words and sentences

Physical interaction

Participants read, write and post in Moodle discussion module.

Interaction

Use the mouse to control the play bar and links

Conditions

Learner can repeat sounds at will

Feedback

Learner can listen to self.

Criteria

Not provided.

Example 3, Objective 2

Objectives

To evaluate instructional design using the framework components

Task

1. Evaluate various examples

2. Discuss advantages and limitations

Conditions

Prior• Participants are interested in and have some experience of the topic.• Participants have been introduced to the framework.

During• Participants can refer to components checklist

Physical interaction

Talking f2f; look at PowerPoint

Feedback

Feedback from presenter and from other participants.

Criteria

Each question in the components checklist has been answered correctly according to the consensus of the group.

Example 4: This presentation

Objectives

To apply the principles of Domain X

Task

1. Read about the principles

2. Discuss the principles verbally and/or in writing

Conditions

Prior• Participants are motivated to learn the principles• Participants have access to the readings laying out the principles

During• Participants can refer to readings

Physical interaction

Participants read, write and post in Moodle discussion module.

Feedback

Feedback from instructor and from other participants.

Criteria

To be determined at a future date

Generic Example: Course X

Conclusion(s)

Does this framework work?

Is this framework useful?

What are its limitations?

Comments/questions?