Enhancing Your Canvas Course · 2) Small Group Evaluation comprising three members from the target audience to assess the effectiveness of the training module. Formative Evaluation

Enhancing Your Canvas Course

Formative Evaluation Report

Instructional Designer: Bryan Hauf

11/28/2016

EDTC 6323 Project 3: Summative Evaluation Report 9/15/2013

Page 1 of 20

Introduction The self-paced e-learning course developed intends to build faculty competency in designing Canvas

courses that facilitate ease-of-use no matter the format. Major themes include: 1) organizing materials

based on learner needs, 2) creating easily navigable course structure and sequence given LMS best

practices and Quality Matters Standards, and 3) developing activities and materials that facilitate and

support learner efficacy using Web usability/accessibility guidelines and Quality Matters Standards.

Two strategies were used to formatively evaluate the course:

1) Subject Matter Expert (SME) review comprising one reviewer to validate content, structure,

appropriateness, and overall design of the learning experience.

2) Small Group Evaluation comprising three members from the target audience to assess the

effectiveness of the training module.

Formative Evaluation Strategy #1: Subject Matter Expert Review

Introduction A subject matter expert review was conducted after completing the development phase of the

instructional solution. A colleague, an instructional designer of 15 years, performed the SME review of

the course given her expert level knowledge in all topics instructed. The review was performed by the

SME without observation over the span of 3 days from her home office.

An oral communication was performed to initiate the review. The following information was delivered

to the SME:

1. A brief introduction of the course.

2. Instructions for the SME to review each lesson, including the pre and post course activities. The

SME does not have to complete activities, but evaluate their appropriateness and quality.

3. Comments and feedback should be provided page-by-page using the template to be furnished

over email.

4. The SME is to review the following in the course:

a. Content quality

b. Content flow and structure

c. Ease of navigation

d. Readability (includes spelling and grammar)

e. Assessment Activities and criteria

f. Learning outcome alignment

g. Visual design of course

An email communication was sent to the SME linking to a document with instructions and a template to

recording findings, feedback, and recommendations. A copy of this document is located in Appendix A.


Page 2 of 20

Summary of Findings The following table organizes the Subject Matter Expert’s comments and feedback/recommendations

and my, the instructional designer’s, response for addressing each item.

Page/Object Feedback/Recommendations Instructional Designer Response

Homepage Video is nicely done and very

cheerful to listen to.

In this sentence, remove the

word above “Click the Start Here

link above when finished to

begin this course.”

Revise sentence as recommended.

Lesson 0: Course Information In this sentence “Directions on

where to go next after reading

can be found on the bottom of

this page” remove the words

after reading.


Lesson 0: Course Structure & Activities Is it a requirement to write your

objectives with the phrase

“Given” for each? I am

wondering if this is the clearest

way to express the outcomes. It

makes perfect sense on Lesson 4,

but not so much on 1, 2, 3. How

about “Using various scenarios”

(because the will be using the

scenarios, correct?);

“Referencing research based

standards” (because they will be

referencing or referring to the

research based standards)

It may not be necessary to call it

a “self-paced web-based e-

learning module”. You could

remove web based. Is “learner-

usable” really a word phrase?

How about user friendly, which

you use later on the page.

Revise sentence to “self-paced elearning

module”.

Replace “learner-usable” with “user-

friendly”.

Revise learning outcomes 1-3 to reflect

recommendation. Will need to update

within each lesson and key assessment.


Page 3 of 20

Lesson 0: Technology & Skill Requirements

Content the course instructor

with Subject Line "Canvas

Practice Course" if you do not

have one. Should be Contact.

need to have = should have

Replace “Content” with “Contact”.

Replace “need to have” with “should

have”.

Lesson 0: Pre-Course Self-Assessment Consider emphasizing “is not” in

Q2 and Q3 so participants do not

misread the “is not.”

Question 2 - Is it missing an

answer option? There does not

appear to be an option that is

“not” a QM standard

requirement - you have three

possible answers and all the

other MC questions have 4.

Answer option is not missing. All are

standards, but the learner should be

looking for the “essential” or “required”

standard. Update question stem to

emphasize.

General Comments on Introduction Module: The page design is easy to read and beautifully laid out. The structure adds to the usability of the content. I like the use of subtle icons and the graduated bands of color headers that do not overpower the content. Is there a purpose for double labeling these items? Is it for clarity? Consider removing ( ) and making the explanatory words part of the Practice Activity (Practice Exercise): Key Assessment (Evaluation): Lesson Wrap-up (Summary):

Remove parenthetical clauses as suggested.


Lesson 1: Overview Same comment use of the word

“Given” in the objective.

Revise learning outcome 1 to reflect

recommendation.

Lesson 1: Best Practices for Organizing Your Course

Should the learning best

practices table have references?

Consider changing this wording:

The file should be identifiable

without having to click for the

content.

“The name of the file should

References are provided below the table

in article. Will explore adding academic

references instead of 2nd

party articles.

Add whitespace if possible, or break

table into two separate lists.

Like how the sentence is currently

worded. No plan to revise.

Add reference to a Faculty Focus article


Page 4 of 20

clearly identify it’s contents.”

Im going to ask for a reference

on this statement on your audio,

not because I disagree, but

because it is a large sweeping

statement.

“Making sure that assignments

are always due on the same day

of the week and modules always

begin on the same day of the

week helps strengthen your

course structure and allows

learners to stay on task through

routine.”

Elearning best practices table -

consider adding white space for

readability.

summarizing the findings from “A Guide

to Creating Modular Courses”.

Add additional reference to University

course shell template design.

Lesson 1: Canvas Tools for Organizing Content and Navigation

On files under exploring Canvas -

is the use of spaces in file names

now acceptable due to

technology improvements? In

Canvas it is OK, but in general is

it still a best practice not to use

spaces?

For readability change the quick

check 1 answers. Do not use

Roman numerals. Use the

terminology as the answer

selections.

Remove roman numerals and include

terminology directly within answer

choices.

Lesson 1: Practice Activity It is great that Ross and Rachel

have moved beyond TV comedy

to course design. :-)

Reminder - if you have an

assessment question, using a

term that is not used in the

content of the course, even if it

seems obvious, is not a

suggested practice. Specifically,

“self-descriptive” is not used in

“Self-descriptive” is mentioned on the

content pages within the comparison

tables. Will not revise.


Page 5 of 20

the content. (q1)

Lesson 1: Key Assessment Revise learning outcome 1 to reflect

recommendation.

Lesson 1: Wrap-up Nice use of infographic as

something learners can take

away from the course.

General Comments on Lesson 1: The use of a contrasting color with the continued same page design adds visual interest. Consider using less ( ) in the text. I would consider changing the wording on activities that are optional. I would call these practice activities (which I think you do later). I think if you call them optional, participants will mostly skip them. Consider saying this: Complete the exercise Lesson 1: Practice Activity. This activity is highly recommended for successful completion of the Key Assessment. So users will not automatically skip them. If you are worried about labeling optional and required, do so in one place - on the “How each Lesson is structured” page call them optional there.

Remove word “optional” from practice activities and only include it in introduction module, as recommended.


Lesson 2: Overview Good use of color again.

Same comment use of the word

“Given” in the objective.


recommendation.


Page 6 of 20

Lesson 2: Canvas Tools for Creating a Custom Homepage

“When students access the

course the first couple of times,

they need a homepage that

conveys the guidelines offered

by Quality Matters.” Suggest re-

writing this sentence. How

about: “When students access

the course the first couple of

times, they need the information

and guidance as suggested by

the Quality Matters guidelines.”


Lesson 2: Practice Activity On the mac, links, for example, in

this: “Taking a look at the

Example #1 Custom Course

Homepage, the longer page,

what recommendations would

you make to the instructor to

improve the page? Provide 2-3

suggestions in a brief narrative

(approx. 3-5 sentences)” the link

is not underlined and the

contrast is not enough to easily

find the link; see screenshot

image.

Adjust contract and force underlines to

make it visible as a link.

Lesson 2: Key Assessment – Develop a Course Homepage

The rubric is very detailed and

complete. Great!

In this sentence the rubric is

double linked: “Using the

Homepage Example #1 and

Homepage Example #2 provided

in this lesson and the grading

rubric.

View in a new window as your

guide.”

In this sentence “Incorporating

strongly recommended

standards is strongly

encouraged.” remove one of the

“strongly”.


recommendation.


Page 7 of 20

General Comments on Lesson 2: Consider adding a story to the course. You have the right idea with the comment below, but the introductions to each module need a bit more warmth. I suggest changing this introduction into some kind of story that flows throughout the course: “Have you ever been given a complex task without guidance on how to begin or complete it?” Turn this into a student story that is used as a “case” throughout the modules. For example: “Today, Annette accessed her Canvas course to complete her course media project.” And then she could not complete this complex assignment because there were not enough details in it, blah, blah. I would even add an image of Annette as a reminder that the users that courses are being organized and designed for are people. Annette looking confused: http://www.webnode.me/confused-looking-person.html You have this kind of warmth in the assessment as where you use names (like Rachel and Ross) in the questions.

Rewrite introductions to each module to add a bit more warmth, as recommended. Use an Avatar to personalize message.


Lesson 3: Canvas Tools for Creating Assessments

Nice use of thinglink and a

previous course asset to

demonstrate the course is

designed using best practices.

Accessibility information on

thinglink should probably be

included.

Accessibility information will be added

where thinglink is used, if it is available.

Lesson 3: Practice Activity In the practice activity Q1,

suggest using alignment since it

is the terminology used in the

previous explanation: “Suggest

that the assessment be revised

since it is not aligned with the

learning outcome.” Mastery

should probably not be used

unless mastery is defined in the

course.

Adjust practice activity to reflect

recommendation for using the word

“alignment”.

Lesson 3: Key Assessment – Create an Assignment

Nice use of video with captions.

The movement in the video adds

interest and gets attention.

Another very complete and

usable rubric, great job.

http://www.webnode.me/confused-looking-person.html

http://www.webnode.me/confused-looking-person.html


Page 8 of 20

Page/Object Comments Feedback/Recommendations

Lesson 4: Overview The more I read the verb

leverage, the more I wonder if it

is measurable. I a slightly lest

subjective verb, even though you

are using a rubric.

“Leverage Canvas tools for

creating usable content and

instruction.”

Change “the same principles and

standards can be applied for

content design.”

the same principles and

standards can be applied to

content design.

Change sentence as recommended.

Lesson 4: Web Design Usability Guidelines

On the bulleted list for Usability

Guideline 1, be consistent with

“.” at end of sentences.

On the quick check, this

statement is unclear: “It is fine to

just accommodate the

accessibility of learners.”

Add periods to make bullet list set

uniform.

Change question stem to

“Accommodating solely accessibility

guidelines makes content usable for

learners.”

Lesson 4: Key Assessment – (Re)-Design a Learning Activity

May want to reconsider how

much is each rating of the rubric

for this assessment. IF each item

is worth 3 points, go ahead and

make each bulleted point a

rating.

Adjust rubric so each bullet point is

worth 3 points each.


End of Course: Course Evaluation Change this for clarity: “Course

activities enhanced my

understanding of the skills that

are described in the course.”

And this

Adjust item 2 in survey to reflect new

statement recommended.

Add “for this learning activity” to the end

of question 9.


Page 9 of 20

“The delivery method used was

appropriate.” appropriate for?

Synopsis All of the recommendations provided by the Subject Matter Expert are revisions that have been

identified as immediately actionable within the course. These revisions include grammar, spelling errors,

and sentence revisions for clarity identified throughout each lesson; adding sources to content within

lesson one; adapting the lesson 4 rubric will also be completed immediately; and updating the story

within the lesson overviews to add warmth.

Formative Evaluation Strategy #2: Small Group Evaluation

Introduction A small group evaluation was conducted after completing the Subject Matter Expert review to gain

feedback from members of the target audience on the effectiveness of the course. Three individuals

with varying backgrounds participated in the evaluation. Participant #1 has a strong education

background with experience teaching online and using e-learning practices. Participants #2 and #3 have

introductory level knowledge of the content presented. Each participant was passively observed in their

work office to document behaviors, struggles, issues, successes, questions, and comments as they

completed assigned lessons. In order to reduce time spent on the evaluation, each participant was only

required to complete a portion of the course:

Participant #1: Course Introduction, Lesson 4

Participant #2: Course Introduction, Lesson 1, Lesson 2

Participant #3: Course Introduction, Lesson 4

Meetings were scheduled 2 weeks ahead of time. Participants were given access to the course 2 minutes

prior to the review. Lessons were assigned at the start of the evaluation. The following information was

delivered to the SME prior to beginning the review:

1. A brief introduction of the course

2. Expectations from participant

a. Participant can ask questions if unable to progress or needs clarification

b. Must complete pre- and post-course assessments and Course Evaluation

3. Lesson to review

4. Instructional designer’s role as a passive observer


Page 10 of 20

Summary of Findings Notes taken by instructional designer are located in Appendix B.

Participant #1

Page/Object Comments Instructional Designer Response

Course Information Learning objectives: move given statement to end of sentence

As per SME review, given statements will be adjusted.

Course Structure page

Practice activity description is not consistent with other bullets.

Make bullet consistent by revising the first sentence to include the word “Using”.

Pre-course Assessment

Answers are visible Adjust setting so answers are not visible; only display score.

Lesson 4 Content Page 1

Before accordion, “is” is missing from the sentence. Emphasized text color in overview text is distracting. Accessibility section: Last two bullets need to start with a verb.

Add “is” to the sentence before accordion. Change color emphasis to a neutral grey color. Change to all directional information in course. Change bullets to start with a verb for consistency.


Remove comma before WordPress. Convert Word documents to PDF.

Remove comma before WordPress. Documents will not be converted to Adobe PDF. Find as Word documents since they are viewable with the Box integration in Canvas.

Lesson 4 Practice Activity

Answer choices don’t appear to be randomized. Links in Quiz activities are not opening in a new tab causing the learners to have to re-enter the quiz from where they left off. Feedback not displaying in Question 3.

Randomize matching answer choices. Adjust links to open in a new tab/window. Add feedback for Question 3.

Lesson 4 Key Assessment

Since learners are self-scoring, may be beneficial to have an interactive rubric for the learners.

Turn Rubric documents into checklists that learners can download and interact with.

Participant #2


Course Dashboard Text is a tad confusing on dashboard. As per SME review, given statements will be adjusted.

Course Information Instructor intro does not indicate instructor’s role. Links to other activities in text is a bit confusing with having the navigation arrows. Purpose of Canvas Practice Course is not clear.

Include role of instructor on page. Modify course navigation instructions to indicate the purpose of linking to items that are delivered in sequence. Clarify instructions and purpose of having a Canvas Practice Course and knowing if you have such a course.


Page 11 of 20

Lesson 1 Content Page #2

List in text before accordion is in a different order than accordion items.

Reorganize list to be in same order as accordion items.

Lesson 1 Practice Activity

“instructions” header is duplicated in quiz. Question 2: Naming convention is not defined in content. Missed that this question was multiple answer.

Remove one of the instructions header across all practice activities. Update content to define naming convention. Emphasize question stem to indicate that learner should select all that apply.

Lesson 1 Key assessment

Change instructions to directions or remove header.

Same issue as practice activities. Adjust so only one “instructions” header is on page.


Instructions for annotations were a bit confusing.

Clarify instructions for how to interact with the ThingLink examples and the key used for annotations.

Participant #3


Course Information Links to other activities in text is a bit confusing with having the navigation arrows. Learning outcomes include the word “faculty”. Comes across as being critical of this audience. “Your First Tasks” is confusing since it links to a task and then progress using the next buttons. Font appears to be compressed. The word “final” before grade indicates a grade is given when the activities are7 not graded.

Modify course navigation instructions to indicate the purpose of linking to items that are delivered in sequence. Replace word “faculty” to “learners” across all outcomes. Do not link items. Just list the first tasks learners are to complete and direct them to click on the “Next” button. Font is standard to the LMS. Will add instructions for how to zoom in using browser functions. Remove word “final” from grade.

Pre-course self- assessment

Question 3 response b. Specific and descriptive criteria is unclear.

Will not change since this is the purpose of the pre-course self-assessment. Learners will know the terms after completing the content.

Lesson 3 Overview Content seems dense and a lot info to consume.

Clarify that learners are not expected to memorize content.

Lesson 3 Content Page #1

Accordion: Sentences are long and complicated.

Review sentences and adjust to be less complex if needed.

Lesson 3 Key Assessment

Time for completing the assessment may be better at the beginning so it is easily identifiable how much time is needed right from the start.

Move time and grading statement to top of the page.

Synopsis All of the issues identified through observing each participant are immediately actionable within the

course. These revisions include fixing grammar errors and typos; improving navigation instructions;

adjusting assessment settings and instructions; adapting content to be less dense and easier to read by


Page 12 of 20

rewriting complicating sentences; improving the usability of pages by adjusting colors of directional

elements. Long-term revisions may be identified after learners complete the course in full without

observation by analyzing the post-course evaluation, pre- and post-course evaluations, and achievement

of learning outcomes in key assessments.

Summary The practicum project allowed me to demonstrate improvement since beginning the educational

technology program by applying instructional design and technology practices final time. By

incorporating many of the lessons learned throughout the program, I was able to analyze the target

audience while also considering the diverse needs of learners, leverage instructional media principles to

provide variety in content delivery, and use integrated course design principles to align assessments and

instruction with the performance outcomes all in the pursuit of developing an effective instructional

solution. But that isn’t to say there weren’t additional lessons to learn.

In progressing through the project, I learned to be realistic with the time available. The breadth of the

course developed was perhaps too much for the time allotted. By having to rush through the

development phase, more time was spent on earlier lessons leaving some gaps in content coverage in

the final lesson. I should have used the opportunity to focus on developing a product that is still a

weakness, such as a self-contained, self-paced course geared towards corporate professional

development. In hindsight, the content may have been better if delivered as separate content packages

instead of one large course given the volume of information presented in each. In the future, I would

like to expand my skillset by learning how to use instructional authoring programs such as Articulate

Storyline and Adobe Captivate to provide me with more work opportunities in the future.


Page 13 of 20

Appendix A

Subject Matter Expert Evaluation Template Directions for Subject Matter Expert

The SME will review the course for the following:

Content quality

Content flow and structure

Ease of navigation

Readability (includes spelling and grammar)

Assessment Activities and criteria

Learning outcome alignment

Visual design of course

Use the table below to record feedback and recommendations on a page-by-page basis.


Homepage

Lesson 0: Course Information

Lesson 0: Course Structure & Activities

Lesson 0: Technology & Skill Requirements

Lesson 0: Pre-Course Self-Assessment

General Comments on Introduction Module:


Lesson 1: Overview

Lesson 1: Best Practices for Organizing Your Course

Lesson 1: Canvas Tools for Organizing Content and Navigation

Lesson 1: Practice Activity

Lesson 1: Key Assessment

Lesson 1: Wrap-up

General Comments on Lesson 1:


Lesson 2: Overview

Lesson 2: QM Standards for Learner-Centered Course Introductions

Lesson 2: Canvas Tools for Creating a Custom Homepage


Page 14 of 20


Lesson 2: Key Assessment – Develop a Course Homepage

Lesson 2: Wrap-up



Lesson 3: Overview

Lesson 3: QM Standards & Best Practices for Assessments

Lesson 3: Canvas Tools for Creating Assessments


Lesson 3: Key Assessment – Create an Assignment

Lesson 3: Wrap-up



Lesson 4: Overview

Lesson 4: Web Design Usability Guidelines

Lesson 4: Canvas Tools for Creating Usable Learning Activities


Lesson 4: Key Assessment – (Re)-Design a Learning Activity

Lesson 4: Wrap-up



End of Course: Post-Course Assessment

End of Course: Course Evaluation

General Comments on End of Course Activities:


Page 15 of 20

Appendix B: Passive Observation Notes

Participant #1


Page 16 of 20


Page 17 of 20

Participant #2


Page 18 of 20


Page 19 of 20

Participant #3


Page 20 of 20