ANALYZING COURSE MANAGEMENT TECHNOLOGY USED FOR ONLINE CLASSES

By:

Erika Sievers

A Senior Project Presented in Partial Fulfillment of the Requirements for the Degree Bachelor of Science

Arizona State University

Polytechnic Campus

February 2018

i

ABSTRACT

The purpose of this research is to discover how to construct the best online

environment for students in higher education. The environment consists of Learning

Management Systems that utilize usability testing. The first stage of the project begins

with surveying students enrolled in the online Senior Project course GIT 480 of the

Graphic Information Technology program at Arizona State University. The second stage

is to analyze documents that had data on Learning Management Systems such as

Blackboard, Canvas, and Moodle. The study found that Blackboard, Canvas, and

Moodle, can each create a positive learning environment for students. The study found

that all universities have a unique culture of education and the features of these three

Learning Management Systems can be customized to fit most. It is important that the

universities conduct ongoing usability testing to ensure that it meets the needs of their

current students.

ii

TABLE OF CONTENTS

CHAPTER PAGE

I. INTRODUCTION…………………………………………………………………… 1

Need for the Project…………………………………………………… 1

Significance of the Project………………………………………….. 2

Statement of the Problem………………………………………….. 2

Limitations of the Study…………………………………………….. 3

Definition of Terms…………………………………………………… 4

Summary……………..…………………………………………………… 5

II. REVIEW OF RELATED LITERATURE……………………………………... 6

Website Usability…………….………………………………………… 6

Course Website Usability……..……………………………………. 7

Learning Management System……………………………………. 8

III. METHODOLOGY…………………………………………………………………… 10

Usability Survey ………….……………………………………………. 10

Documentary Analysis …………………………………...…………. 10

IV. RESULTS OF THE PROJECT…………………………………………………… 14

Usability Survey ………..……………………………………………… 14

Documentary Analysis ………………...……………………………. 20

V. CONCLUSIONS AND RECOMMENDATIONS…………………………….. 21

Conclusions………………………………………………………………... 21

Recommendations……………………………………………………… 22

REFERENCES………………………………………………………………………………….. 23

iii

APPENDIX………………………………………………………………………………………. 26

Usability Survey …………………………………...…………………… 26

iv

I. Introduction

Need for the Project

Blackboard Classroom is a Learning Management System, “used in 75

percent of colleges and universities and more than half of K-12 districts in the U.S.

With more than 20 million users across 20,000 organizations, the 16-year-old

company remains the dominant player in the market, serving both professional and

academic sectors” (Epsom, 2014). When taking online courses, the average student

will often analyze the usability and simplicity of their online classroom without

thinking about it. They may become frustrated at the features that the platform

offers or does not offer. They may also run into errors that should be avoided while

participating in an academic course.

Online learning enables students to study at their own pace in the comfort of

their own home. It is a great resource for students who cannot make the time to

travel to school. These students must be able to perform tasks on their own, and

receive assistance when needed, in a timely manner. There are many alternative

Learning Management Systems that must be taken into account when the college or

university is selecting the right system.

The goal of an online classroom is to allow students to experience an

environment similar to an in-person classroom. Thus, ensuring a positive user

experience and the ease of learning are the most important aspects of selecting an

online environment for students. This project will determine whether Blackboard

Learn is the best option for and universities to choose as their virtual online

v

classroom. This project will benefit the users of Blackboard Learn, both student and

teacher, as a better online classroom experience can be expected.

Significance of the Project

The idea of the perfect online classroom is of vital importance in the selection

of a Learning Management System. This system allows students and teachers to

communicate and interact with each other. These systems also provide schools with,

“tools and techniques that make for better cost efficiency and results effectiveness,

even for client-responsive programs” (Shah, 2017). Since there are currently many

platforms available for colleges and universities to choose from in the market, it is

difficult to understand which platform will be the best resource to help both

students and staff succeed. Analyzing the usability of the platform as well as taking

surveys from students will help to determine which platform is the best choice for

online schooling. This will provide insight into what Blackboard is successful or

unsuccessful with.

Statement of Problem

The objective of this research is to analyze and determine if Blackboard is the

best course management system to host online courses for colleges such as Arizona

State University. This research will showcase both the advantages and the

disadvantages of Blackboard and will use these points to advance the functionality

of this software. These five specific objectives will aid in creating an avenue for

research:

vi

1. Is Blackboard an effective course management system for online students?

2. What resources are needed to improve the user experience with Blackboard?

3. Is there a usability problem that exists with Blackboard or with the instructors?

4. What are the available alternatives to Blackboard?

5. What are the advantages and disadvantages of using Blackboard over their competitors such as Canvas?

Limitations of the Study

Although this research project has reached its aims and objectives, some

unavoidable limitations arose. Due to time and resource constraints, such as the

inability to send out a usability survey campus-wide, research populations are small.

The survey and study will focus exclusively on 14 Blackboard Learn users who are

attending the online course GIT 480 at Arizona State University in the United States.

Usability testing typically involves randomly selected participants of all computer

experiences, ages, genders, and backgrounds. The participants in this study will all

be familiar with Blackboard Learn and there is not much variety with experiences

on other Learning Management Systems. This online course only lasts seven and a

half weeks which is not enough time for the researcher to collect and analyze

usability surveys from every student in the class. The length of the class also does

not allow enough time to have the students participate in every test that is typically

involved in extensive website usability testing. It would be best to have both

students and faculty complete a true usability test by performing tasks on the

Learning Management Systems.

vii

Definition of Terms

Accessibility - Features that increase software usability for users with certain

impairments (“Accessibility”).

Feedback – Notifying the user about any successes or failures as a result of their

input or actions.

Information Architecture – This is the focus on “organizing, structuring, and labeling

content in an effective and sustainable way” (Information Architecture

Basics, 2013).

Learnability – “interfaces that don’t require instructions or even a long process of

trial and error to figure them out” (Principles of Website Usability, 2018).

Learning Management System (LMS) – A software application used to organize and

distribute e-learning materials, assignments, and assessments; track and

calculate grades; and facilitate communication among students and teachers

(Learning Management System).

Navigation – The route that a website uses to help the user travel from one page to

another page of the website.

Usability – Ensuring that the user can accomplish certain tasks that are presented

on a website.

User Experience (UX) – “The perception and response of a person toward design

elements of software or digital media while interacting with it” (“User

Experience”).

viii

Summary

This research project will focus on the usability of Learning Management Systems. It

will center on on how other researchers have determined usability for online

courses and how current Arizona State University students, enrolled in GIT 480,

view the usability of Blackboard Learn. The research will demonstrate the strengths

and weaknesses as seen through the eyes of actual users.

ix

II. REVIEW OF RELATED LITERATURE

Website Usability

Steve Krug, information architect and user experience professional,

published the book Don’t Make Me Think: Revisited, in 2013. The title of the book

explains the basis of the author’s message; in fact he calls it, “Krug’s First Law of

Usability” (Krug, 2014). He describes this law as, “the overriding principle – the

ultimate tie breaker when deciding whether a design works or doesn’t” (Krug,

2014). Usability is about ensuring that websites, as well as any features that are

included, work well and that the average person with minimal experience could use

it without experiencing frustration or error. In other words: the website should

make things as obvious as possible for the user to avoid confusion or tedious steps

that may waste the user’s time while completing tasks. “Unlike interviews or focus

groups that attempt to get users to accurately self-report their own behavior or

preferences, a well-designed user test measures actual performance on mission-

critical tasks. If the user cannot figure out how to complete a purchase, no amount of

“but I really like the website!” comments are going to make up for it” (“Usability

Testing”, 2018).

“When conducting user testing, the researcher reads a participant one task at

a time, such as “Find out how to contact technical support,” and allows the

participant to complete the task without any guidance. To prevent bias, the

researcher follows the same “script” when explaining the task to each participant”

(“Usability Testing”, 2018).

x

In an article titled, “A Case Study on Reliability and Usability Testing of a Web

Portal”, the authors discussed web portals. The article claims that “Reliability testing

of web portals is distinctive as compared to that of conventional software. It

requires usability and reliability metrics besides functional testing to ensure diverse

users are able to use such portals without any formal training and can rely on the

portals to meet users’ needs or requirements quickly and efficiently” (Ali-Shahid &

Sulaiman, 2015). Universities have diverse users that are often not given formal

training on successfully navigating the portal. Courses often have due dates for

assignments, projects, and assessments. It is imperative that the student user is able

to locate information easily. “By pairing good usability testing and good web design,

the result is a website that incorporates feedback from real users and increases

users satisfaction” (Dishman, 2015).

Course Website Usability

Usability is an issue when choosing Learning Management System for online

education. According to an article on the University of Pittsburg, “An Introduction to

Usability and your Blackboard Course Site, “The “usability” of a website refers to

how easy a site is to use for the people visiting it (“An Introduction to Usability”,

2017). In the case of Blackboard, the visitors are the students enrolled in a

particular course, so usability is now focused on a specific group. The article

emphasizes that, “Whether you just use Blackboard to deliver course readings to

your students, or are teaching a partially or fully online course, the issue of how

easy it is for students to find and access what they need inside your course shell is

xi

an extremely important one” (“An Introduction to Usability”, 2017). “Usability

research helps organizations understand user needs, identify potential issues, and

generate ideas for improvement” (“Testing Content on Wbsites”, 2015).

The author acknowledges that many of the features in Blackboard are set in

stone, but that there are features that can be modified within the course. It is

important that the course is well-organized. The course builder needs to ensure that

the course is well-organized, has useful menus, and the content is properly named

to make it easier to be found (“An Introduction to Usability”, 2017). Students tend to

become frustrated when there are many steps to locate pertinent information, so

ease of searching is essential to an overall positive experience in a course.

“In recent years, the number of low-cost or free testing tools has increased,

leading to new opportunities for colleges to evaluate their websites (Dishman,

2015). These tools will help colleges analyze their sites for best usability. They are

able to evaluate the websites by distributing usability tests to users of all different

experience levels.

Learning Management Systems

What is a Learning Management System? Haitham Khedr, founder and CEO of

HK Consultancy, answers that question. He stated in an article on

eLearningindustry.com that, “A Learning Management System (LMS) is a software

application for the administration, documentation, tracking, and reporting of

training programs, classroom and online events, eLearning programs, and training

content (Khedr, 2017). Most colleges and universities utilize the eLearning

xii

programs. The Learning Management programs not only take the place of the

traditional classroom, but it also enables the course directors to track and analyze

data within their course shell.

Khedr also notes that, “It is also used by educational institutions to enhance and

support classroom teaching and offering courses to a larger population of learners

across the globe” (Khedr, 2017). Arizona State University and many universities

across the country are able to offer classes to students that do not live near the

campuses.

The use of a Learning Management System brings more learning

responsibility to the student. The role of the faculty member transforms from

teacher to facilitator (Chiasson, Terras, & Smart, 2015).

xiii

III. METHODOLOGY

Usability Survey

To provide information on how well students interact with Blackboard

Learn, a series of questions concerning experiences with Blackboard Learn such as:

comparisons to competitors, ease of learning, ease of navigation, frequently used

features, most useful features, and platform consistency. This questionnaire was

created on Google Forms and emailed to the entirety of the online class GIT 480 at

Arizona State University. The students were required to answer all questions

honestly to evaluate their opinion on Blackboard Learn.

Documentary Analysis – Blackboard vs Canvas vs. Moodle

FinancesOnline is an independent review site that helps organizations to

make decisions about their technology essentials. This site compared many of the

Learning Management Systems that are currently available. The images below show

the features that they present for Blackboard, Moodle, and Canvas. Listed are

twenty-one features for Canvas, sixteen for Blackboard, and twenty for Moodle. The

amount of features is not necessarily important to an organization. It is more

important that the Learning Management System contains the features that best fit

the needs of the organization. The ability to upload audio and video appear to be a

stand-out feature of Canvas. The article concludes that “The main goal behind

Blackboard is to create and deliver online courses and learning successfully to

students with few or no face-to-face meetings” (Reviews, 2017). This particular

xiv

feature could persuade an organization to choose this Learning Management System

over the other two in the following comparison chart.

Note. Adapted from Blackboard Reviews: Overview, Pricing and Features. (n.d.). Retrieved February 15, 2018, from https://reviews.financesonline.com/p/blackboard-learn/#features

Documentary Analysis – Blackboard vs. Canvas

Emory Libraries and Information Technology, at Emory University in

Georgia, compared Canvas with Blackboard. It was discovered that “Canvas is

designed to deliver a simple, transparent, and consistent online learning experience

for students. Blackboard emphasizes instructor control and customization. This

xv

contrast is reflected in the facts that Canvas (a) allows students to access areas by

default that Blackboard hides by default or calls for more clicks to find, (b)

standardizes navigation menus across the courses, (c) organizes content in a more

extended, less hierarchical style for quicker access, and (d) allows access to the

same course content differently in several tools so students can choose the view

they prefer” (Emory Libraries, 2016). It appears that Canvas is more geared to be

intuitive for the student. This could lead to a better user experience for the student.

As learning styles are shifting, it may be beneficial for an instructor to give up some

control of the course shell so that the students can manipulate the course to better

suit their educational needs.

Documentary Analysis – Blackboard vs. Moodle

iSpring Solutions, Inc. compared the features of Blackboard and Moodle. The

article on their website summarizes that “Moodle is a powerful free solution, but

tailoring it for your specific needs may require many hours of programming.

Blackboard is fully loaded, but expensive (Moodle vs. Blackboard, 2017). Moodle

may be the Learning Management System of choice for cash-strapped colleges. The

article states that Moodle “is one of the most popular LMSs, mostly used by

institutions with between 1,000 and 2,000 full-time students” (Moodle vs.

Blackboard, 2017).

The comparative chart below illustrates that Blackboard has many more

features than Moodle. It is probable that many colleges do not need the extra

features provided by Blackboard.

xvi

Note. Adapted from Moodle vs BlackBoard – That is the Question. (2017, June 21). Retrieved February 16, 2018, from https://www.ispringsolutions.com/blog/moodle-vs-blackboard/

IV. Results of the Project

xvii

Usability Survey

There were a total of 16 anonymous survey respondents. This survey

remained open from February 7, 2018 to February 14, 2018. The results of this

survey included all online students who were between experienced and very

experienced with Blackboard Learn. There were also some participants who have

used other Learning Management Systems in their previous academic years. The

majority of participants seem to enjoy Blackboard Learn, but have acknowleged that

it does need some improvements and lacks consistency. A couple of participants

have expressed their preference for Canvas, another Learning Management System.

xviii

xix

xx

xxi

xxii

xxiii

Documentary Analysis

The results of the documentary analysis demonstrate that Blackboard is an

effective course management system for online students. The features are

comparable to the alternative choices. Every university appears to have its own

unique culture and the Learning Management System needs to be able to adapt to

that culture. Blackboard is customizable to the degree that it can meet the needs of

many universities.

In comparing Blackboard, Canvas, and Moodle, the usability of these systems

is dependent upon the University. The platforms can be customized for increased

usability. The universities must make due diligence to conduct usability tests of

their particular version of the Learning Management System.

All three Learning Management Systems researched in this project convey

ownership of learning directly to the student. The faculty members must take on the

role of facilitator of the students’ learning.

xxiv

V. Conclusions and Recommendations

Conclusions

The study conducted for this project showcased several different perspectives

related to the research objectives. The following conclusions were reached for the

project’s objectives.

1. Blackboard is an effective course management system for online students. It has many features that are designed to be customized to best suit the educational organization. Students tend to prefer Blackboard over Canvas and Moodle.

2. Usability testing is needed to improve the user experience with Blackboard. Many useful features are available, but need to be adjusted for the students to be able to successfully perform their online activities. Feedback from frequent users would help to adjust the system to fit the needs of each student.

3. The usability problem appears to result from the instructors’ designs of the course within the course shell. Faculty members must take the time to reconstruct their courses to enhance usability. Consistency is necessary to prevent students from relearning the site map for each class.

4. The available alternatives to Blackboard include systems such as Canvas and Moodle. There are a vast amount of Learning Management Systems available, but these three systems are the most used in universities.

5. Blackboard has an initial disadvantage over Moodle because Moodle is a free Learning Management System. This drawback seems to diminish as Universities realize the extra amount of money required to support the Moodle system in the Informational Technology sector. Blackboard has the advantage over Canvas because it has been around much longer and already cemented in universities across the globe. The advantages of Canvas must outweigh the cost of removing Blackboard and starting from scratch. This research shows that Blackboard and Canvas are very comparable in their features and pricing.

xxv

Recommendations

1. Universities should conduct ongoing usability testing of their selected

Learning Management System. Feedback from the actual users is of utmost

importance.

2. Usability tests must be an actual performance test of the website. The

performance test must be conducting identically for all participants. The

results of the performance test should be analyzed and necessary changes

should be made to the Learning Management Systems. If the Learning

Management system cannot meet the needs of the students, a new system

should be implemented.

3. Blackboard Learn must showcase consistency in navigation. Teachers are

given too much freedom in deciding where they can place the content

necessities for the course. Students should not have to relearn the site

navigation for each class they are enrolled in. Universities should receive

feedback from students on which navigation links they feel are the most

important and ensure that every teacher’s site is formatted the same.

4. To improve this study, it is recommended that an actual usability test be

given to students currently enrolled in online classes. There should be a

separate test for each Learning Management System, and students must

actually perform required tasks and give feedback.

xxvi

REFERENCES

“Accessibility.” Accessibility dictionary definition | accessibility defined,

www.yourdictionary.com/accessibility.

Ali-Shahid, M., & Sulaiman, S. (2015). A case study on reliability and usability testing

of a Web portal. Software Engineering Conference (MySEC), 2015 9th Malaysian,

31-36.

An Introduction to Usability and Your Blackboard Course Site. (n.d.). Retrieved

February 10, 2018, from https://www.etskb-fac.cidde.pitt.edu/blackboard/an-

introduction-to-usability-and-your-blackboard-course-site/

Blackboard Reviews: Overview, Pricing and Features. (n.d.). Retrieved February 15,

2018, from https://reviews.financesonline.com/p/blackboard-learn/#features

Chiasson, K., Terras, K., & Smart, K. (2015). Faculty perceptions of moving A face-to-

face course to online instruction. Journal of College Teaching & Learning

(Online), 12(3), 321. Retrieved from http://login.ezproxy1.lib.asu.edu/login?

url=https://search-proquest-com.ezproxy1.lib.asu.edu/docview/1696895692?

accountid=4485

Dishman, M. (2015). Technology Review: Website Usability Testing Software--

Improving User Experience and Satisfaction with Community College

Websites. Community College Enterprise, 21(1), 84-87.

Emory Libraries and Information TechnologyCanvas at Emory. (n.d.). Retrieved

February 15, 2018, from

http://canvas-support.emory.edu/training/compare.html

xxvii

Empson, R. (2014, January 16). Education Giant Blackboard Buys MyEdu To Help

Refresh Its Brand And Reanimate Its User Experience. Retrieved February 19,

2018, from https://techcrunch.com/2014/01/16/education-giant-blackboard-

buys-myedu-to-help-refresh-its-brand-and-reanimate-its-user-experience/

Information Architecture Basics. (2013, October 08). Retrieved February 18, 2018,

from https://www.usability.gov/what-and-why/information-architecture.html

Khedr, H. (2017, September 19). In And Beyond The Classroom, Which Learning

Management System To Choose? Retrieved February 11, 2018, from

https://elearningindustry.com/in-and-beyond-the-classroom-learning-

management-system-choose

Krug, S. (2014). Dont make me think, revisited: a common sense approach to web

usability. Indianapolis, IN: New Riders Publishing(IS).

Learning management system. (n.d.). Retrieved February 23, 2018, from

http://www.dictionary.com/browse/learning-management-system

Moodle vs BlackBoard – That is the Question. (2017, June 21). Retrieved February

16, 2018, from https://www.ispringsolutions.com/blog/moodle-vs-blackboard/

Principles of Website Usability | 5 Key Principles Of Good Website Usability. (2018,

February 19). Retrieved February 23, 2018, from

https://www.crazyegg.com/blog/principles-website-usability/

Shah, H. (2017, September 19). Why Learning Management System Is Useful. Α

Retrieved February 21, 2018, from https://elearningindustry.com/learning-

management-system-useful

xxviii

Testing Content on Websites. (2015, April 26.). Retrieved February 12, 2018, from

https://www.nngroup.com/articles/testing-content-websites/

Usability Testing. (2018). Retrieved February 12, 2018, from

http://www.usabilityfirst.com/usability-methods/usability-testing/

User experience. (n.d.). Retrieved February 20, 2018, from

http://www.dictionary.com/browse/user-experience?s=t

xxix

APPENDIX

Usability Survey

xxx

xxxi

xxxii

xxxiii

xxxiv

xxxv

xxxvi