The Eleventh International Conference on eLearning for Knowledge-Based Society, 12-13 December 2014, Thailand

11.1

Quality Frameworks and Standards

in E-Learning Systems Moncef Bari

1

and Rachida Djouab2

Departement of Didactics,

Université du Quebéc à Montréal, Canada 1bari.moncef@uqam.ca

2djouab.rachida@uqam.ca

Abstract - The e-Learning systems are one

of the areas that have proven their own

importance in promoting the use of IT

technologies in academia environment.

However, the growing need for such

systems requires the adoption of a quality

design approach based on a recognized

quality frameworks and standards

comprising sets of well defined criteria.

This paper investigates the existing quality

frameworks (SCORM, IEEE P1484 and

IMS Global Learning Consortium) and

standards (ISO/IEC 19796, Open

ECBCheck and ISO 9126) dedicated to the

e- Learning systems.

Keywords - Software Quality, E-Learning,

E-Learning Quality, Standards, Frameworks

I. INTRODUCTION

Quality is most often defined as “fitness

for purpose” that is related to the needs of

the user or that the ‘product comply with

defined requirements’ (Rekkedal, 2006). Over

the years, different definitions have been

attributed to the term “quality”. To Crosby,

1979 it is “conformance to user requirements”.

Watts, 1989 refers to quality as “achieving

excellent levels of fitness for use”, while

IBM coined the phrase “market-driven

quality” which is based on achieving total

customer satisfaction” (Abran et al. 2004).

In April 2002, a web-based survey was

carried out in five European languages

(English, French, German, Spanish, and

Italian) over 450 respondents to analyze

important questions about the quality of

e-Learning products and services. The survey

was distributed through the European

Training Village the website of the European

Commission's Agency for Vocational

Training. Respondents were from all EU

countries and the results of the survey (Massy,

2002) shows that:

61% of all respondents rated the quality of e-Learning negatively - as ‘fair’ or ‘poor’;

The two most important criteria for evaluating quality in e-Learning are that it

should ‘function technically without problems

across all users’ and have ‘clearly explicit

pedagogical design principles appropriate to

learner type, needs and context’;

A very small number (4) of respondents

from 3 different countries gave an excellent

rating.

In 2004, the European survey has been

carried out among 5,023 people and 1,407

completed it (Ehlers et al. 2005). The

questionnaire deals with five blocks of topics:

(a) e-Learning in general, (b) quality in

e-Learning in general, (c) use/implementation

of quality instruments in e-Learning, (d)

experience with quality instruments and

approaches, and (e) issues of

statistics/demography.

The study has shown that quality plays a key role in the success of e-Learning and

learning in general, and among the most

important observations, the following have

been noticed:

Quality Frameworks and Standards in E-Learning Systems

The Eleventh International Conference on eLearning for Knowledge-Based Society, 12-13 December 2014, Thailand

11.2

Learners must play a key part in determining the quality of e-Learning

services: a learner orientation is imperative in

the area of quality;

Culture of quality in education and

training: quality development must become

a core process for educational organizations;

Quality must play a key role in education policy;

Quality development as the norm in the educational landscape;

Quality services might be created;

Open quality standards must be widely

implemented;

The paper is organized as follows:

section 2 is dedicated to the quality standards, section 3 describes the quality

frameworks and section 4 presents our

conclusion.

II. QUALITY STANDARDS

In 2005, the new quality standard for

learning, education, and training, ISO/IEC

19796-1, was published and aimed at

helping educational organizations to develop

quality systems and to improve the quality

of their processes, products, and services

(ISO/IEC, 2005). The standard serves as a

reference supporting adaptation to the

specific requirements of the organization. It

includes process description (e.g. evaluation

of didactic methods), methods (e.g.

identification, alternatives, and priorities),

objectives (e.g. adequate selection of one or

more didactic concepts), target group

(competencies and learning styles),

organization, relations, etc. (Rekkedal, 2006).

However, since the standard is a reference

model, it has to be adapted to the needs of

an organization. Pawlowski, 2007 suggests a

quality adaptation model to successfully

implement the standard in educational

organizations and to support the variety of

involved actors.

In 2010, an international quality standard

for e-Learning programs - “Open

ECBCheck (e-Learning in Capacity

Building)” - was officially released (last

version: ECBCheck, 2012). ECBCheck is an

accreditation and quality improvement

scheme for e-Learning programs that

supports organizations in measuring the

success of their programs and allows for

continuous improvement though peer

collaboration. More than 40 international,

regional, and national capacity-development

organizations participated in the development

of this standard. ECBCheck provides a set of

quality criteria to assess e-Learning program

design, development, management, delivery

and evaluation, as well as the quality of

learning materials, methodology, media,

technology and e-tutoring. These quality

criteria are: 1) Information about

organization of program; 2) Target Audience

Orientation; 3) Quality of Content; 4)

Program/ Course Design; 5) Media Design; 6)

Technology and 7) Evaluation & Review.

Chua and Dyson, 2004, proposed ISO

9126 as a tool to evaluate the e-Learning

systems for teachers and educational

administrators. Their research into evaluating

e-Learning systems aims at assisting

educators in evaluating the quality of the

system and incorporates frameworks to

support decision making regarding review of

existing systems and the purchase of new

ones. They also proposed enhancements of

the model by extending the usability

characteristic with additional quality

attributes such as: consistency, simplicity,

legibility, and color use and by including

the “user satisfaction” as a global

characteristic of the model. Furthermore, they

state that the ISO model does not specify the

particular teaching and learning activities

needed for good learning and proposed for the

software developers without educational

expertise a checklist of tools and attributes to

gain an efficient course management.

Padayachee et al. 2010 used the ISO 9126

model for selecting generic external systems

quality characteristics and sub characteristics

for user evaluation of course management

system (Course management systems are

Moncef Bari and Rachida Djouab

The Eleventh International Conference on eLearning for Knowledge-Based Society, 12-13 December 2014, Thailand

11.3

interactive systems that enable educators,

with minimal technology expertise to design,

develop and deliver e-Learning content as

well as measure the outcome of e-Learning

courses). The objective of the research study

was to validate the effectiveness of the ISO

9126 quality model for evaluating the

quality of e-Learning systems. Authors

suggest creating domain specific quality

criteria that relate to selected characteristics

and sub-characteristics of the ISO model.

The e-Learning quality criteria corresponding

to the first four quality characteristics of ISO

9126 were developed to aid in evaluating

users’ view of quality for e-Learning systems

and to obtain feedback from educators in

higher education using questionnaires and

interviews. Furthermore, ISO 9126 has been

customized to identify acceptance criteria and

evaluate a B2B (business to business)

application (Behkamal et al. 2008). Authors

reviewed and compared the existing quality

models, defined the B2B electronic

commerce and described the customization

approach of the ISO 9126 to the B2B

applications. The quality characteristics of

B2B applications were determined and new

quality characteristics have been identified

and added to the ISO model. These

characteristics have the highest ranking and

significance in B2B applications.

On the other hand, The ISO 9126 was

used to evaluate the mobile learning by using

additional metrics that are (Parsons and Ryu,

2006):

Metaphor: does the learner have an overall vision of the learning process?

Interactivity: is the learner able to interact with other users and/or tutors?

Learning content: does the learner feel

that the learning component is of high quality?

TABLE I

QUALITY STANDARDS

FOR THE E-LEARNING SYSTEMS

Standards

for eLearning Description

ISO/IEC 19796

(ISO/IEC, 2005)

ISO/IEC 19796-1 is a quality

standard following the principles of quality management developed

for learning, educa t ion and

training in general and it has been

adopted to the specific needs of developers and providers of online

services and digital resources in

many imp lementa t ions and

projects;

The ISO/IEC 19796-1 standard

was developed by the Working

Group 5 “Quality Assurance and

Descriptive Frameworks” of the s t andardiza t ion commit t ee

ISO/IEC JTC1 SC36;

The quality standard contains the

reference p roces s model

“Reference Framework for the Descr ip t ion of Qual i t y

Approaches” (RFDQ) to help

s t akeholders in learning ,

education, and training and especia l ly in e-Learning or

blended learning to document and

( re- )def ine their everyday

processes;

The reference process model of

ISO/ IEC 19796 -1 is the

integration of two main reference

models-generic process model and

generic description model;

The reference process model

covers the whole lifecycle of the

needs ana lys is , des ign ,

development, realization and

eva lua t ion of any learning opportunity or process including

e-Learning and blended learning.

Open ECBCheck

(ECBCheck, 2012)

Is a new certification and quality

improvement scheme for

e-Learning courses and programs

in international Capacity Building;

It supports capacity building

organizations to measure how successful their e-Learning

programs are and allows for

continuous improvement through peer collaboration and bench

learning;

The ECBCheck tool can also be

used for internal quality check of

the courses and programs.

ISO 9126

(ISO/IEC, 1999)

Was proposed as an international

standard for software quality

measurement in 1992. It is a

derivation of the McCall model.

The most common used quality

standard model. There are several

Quality Frameworks and Standards in E-Learning Systems

The Eleventh International Conference on eLearning for Knowledge-Based Society, 12-13 December 2014, Thailand

11.4

others though, such as IEEE 1061.

Has 3 levels of qua l i t y

characteristics, sub-characteristics,

and attributes.

Defines 21 attributes that a quality

product must exhibit. The 21 attributes are arranged in six areas:

functionality, reliability, usability,

efficiency, maintainability and

portability.

Has identified three models of

software product quality (internal

quality, external quality and

quality in use).

ISO 9126 has been used to

evaluate the e-Learning systems for teachers and educational

administrators to support decision

making regarding review of

existing systems and the purchase

of new ones (Chua and Dyson,

2004).

ISO 9126 model has been used for

selecting generic external systems

quality characteristics and sub characteristics for user evaluation

of course management system

(Padayachee et al. 2010).

III. QUALITY FRAMEWORKS

There are a various frameworks for

software developers to improve the quality of

the e-Learning systems they are developing.

(Chua and Dyson, 2004) listed three

frameworks:

The IEEE Learning Technology Standard Committee (LTSC) reference model,

IEEE P1484.1 LTSA which is focusing on

reusability and portability attributes,

The Sharable Content Object Reference

Model (SCORM) that supports content

compatibility (the portability of content from

one e-Learning system to another) and the re-

usability of learning objects,

The Instructional Management Systems (IMS) project that defines technical

specifications to ensure interoperability

between e-Learning systems (IMS Global

Learning Consortium).

TABLE II

QUALITY FRAMEWORKS DEDICATED

TO E-LEARNING SYSTEMS

Frameworks

for the

eLearning

Description

SCORM (ADL,

2001; SCORM, 2004)

The Sharab le Content Object

Reference Model.

Is an XML-based framework

used to def ine and acces s

information about learning objects so they can be easily shared among

different learning management

sys t ems (Birnbaum,1989).

SCORM was developed in

response to a Unit ed S ta t es Department of Defense (DoD)

ini t ia t ive to p romote

standardization in e-Learning.

1997 Advanced Dis t r ibuted

Learning (ADL) Ini t ia t ive network has been formed to create

a way to make learning content

portable across various systems.

ADL created the first version of

SCORM, which originally stood for Shareab le Courseware

Object Reference Model.

It was des igned to facilitate

moving course content and related

information (such as student records) from one platform to

another, to make course content

into modular objects that can be

reused in other courses, and to enable any LMS to search others

for usable course content.

IEEE P1484

(IEEE, 2000; Weibel and

Koch. 2000)

IEEE Learning Technology

Standard Commit t ee (LTSC)

reference model;

Established in December 1996 by

the organiza t ions : IEEE , Microsof t , Edutool , Kutz &

Assoc. , Osaka Univers i t y ,

Carnegie Mellon, MITRE and US

Army;

First standard released in June

2002;

This standard specifies the syntax

and semantics of Learning Object

Metadata, defined as the attributes

required to ful ly/adequately describing a Learning Object.

For this S t andard, a learning

object is defined as any entity--

digital or non- digital-- that may

be used for learning, education or training.

Examples of Learning Objects

include multimedia content,

instructional content, learning

objectives, instructional software and software tools, and persons,

Moncef Bari and Rachida Djouab

The Eleventh International Conference on eLearning for Knowledge-Based Society, 12-13 December 2014, Thailand

11.5

organiza t ions , or event s referenced during technology-

supported learning.

Examples of t echnology-

suppor t ed learning include computer -based t ra ining

systems, interactive e-Learning

environment s , int e l l igent

computer -a ided ins t ruct ion sys t ems , dis t ance learning

sys t ems , and col labora t ive

learning environments.

IMS

(Instructional

Management

Systems)

Learning

Consortium

(IMS 2003; IMS

2004)

Non-profit alliance of suppliers,

higher educa t ion institutions,

school districts and states working

together to develop and bring to market open standards t ha t

make educational applications,

Global content and data plug and

play;

IMS came into existence as a

p roject within the National

Learning Infrastructure Initiative

of EDUCAUSE.

While IMS got its start with a

focus on higher education, the specifications published to date as

well as ongoing projects address

requirements in a wide range of

learning contexts, including of course K-12 schools and

corpora t e and government

training;

he cr i t ica l aspect s of

interoperability in the learning markets have been established by

IMS. Based on these

requirement s , t he draf t specifications (Learning Tools

Interoperability (LTI)) has been

developed outlining the way

software must be built in order to meet the requirements;

he LTI aim to establish a standard

way of integrating rich learning

applications with p la t forms

l ike learning management sys t ems , por t a ls , or other

educational environments;

n LTI these learning applications

are called Tools (delivered by

Tool Providers) and the LMS, or p la t forms , a re ca l led Tool

Consumers.

One of the important drawbacks of these

standards is that they focus on technical

aspects of e-Learning systems and do not

include the usability aspect (how the user will

interact with the system) and are specially

designed for technical trained system

developers (Klasnic, Lasic-Lazic and Seljan

2010). Furthermore, they are too complicated

for the average educators or educational

administrators to understand and apply when

choosing an e-Learning system (Chua and

Dyson, 2004).

IV. CONCLUSION

This paper described the existing quality

frameworks and standards developed for the

e-Learning systems. Based on these studies,

one can say that the technical quality

frameworks are difficult to understand and

apply by the average educators when

choosing an e-Learning system. Indeed, there

is a need to develop instruments and

techniques to be applied by all involved

actors of the organization. In addition,

ISO/IEC 19796-1 is a promising quality

standard but should be adapted to the specific

needs of the developers and providers of the

online services and digital resources. It

appears that there is a need to make further

research to bring this abstract standard into

practice and implement it successfully in the

educational organizations.

On the other hand, the ISO 9126 quality

model is the most applied in the software

engineering community for the evaluation of

their software systems. It is easy to understand

and to apply but needs additional quality

characteristics related to the pedagogy and

design of the e-Learning systems. The Open

ECBCheck standard designed several distinct

criteria areas to assess courses or programs

(ECBCheck, 2012) that could be combined

with the ISO 9126 quality model to improve

the course management.

Our current research is focusing on further

investigations to customize the ISO 9126

model to the e-Learning domain and to build

an integrated quality evaluation tool for the

e-Learning systems.

Quality Frameworks and Standards in E-Learning Systems

The Eleventh International Conference on eLearning for Knowledge-Based Society, 12-13 December 2014, Thailand

11.6

REFERENCES

(Arranged in the order of citation in the

same fashion as the case of Footnotes.)

[1] Abran, A., Moore, J.W., Bourque, P.,

Dupuis, R., and Tripp, L. (2004),

“Guide to the Software Engineering

Body of Knowledge”. IEEE Computer

Society/ ISO/IEC JTC1 SC7. Latest

version downloadable from

<www.swebok.org>.

[2] ADL. (2001), Advanced Distributed

Learning, Sharable Content Object

Reference Model (SCORM), “The

SCORM Run-Time Environment”.

<http://xml.coverpages.org/SCORM-12-

RunTimeEnv.pdf>. Accessed March

2014.

[3] Behkamal, B., Kahani, M., and Akbari,

M.K. (2008), “Customizing ISO 9126

quality model for evaluation of B2B

applications”. Information and Software

Technology. 51, pp. 599-609.

[4] Birnbaum, R. (1989), “The Quality

Cube: how college presidents assess

quality in: Quality in the Academic:

Proceeding from a national

symposium”. National Center for

Postsecondary Governance and Finance.

University of Maryland.

[5] Chua, B.B. and Dyson, L.E. (2004),

“Applying The ISO 9126 Model To The

Evaluation Of An E-Learning”, in

Proceedings of the 21st ASCILITE

Conference, pp. 184-190.

[6] Crosby, P. (1979), “Quality is Free”.

New York: McGraw-Hill. ISBN 0-07-

014512-1.

[7] ECBCheck. (2012), “OpenECBCheck

Quality Criteria”.

<http://cdn.efquel.org/wp-

content/blogs.dir/4/files/2012/07/OpenE

CBCheck-Quality-Criteria-2012.xlsx>.

Accessed March 2014.

[8] Ehlers, U-D., Goertz, L., Hildebrandt,

B., and Pawlowski, J.M. (2005),

“Quality in e-Learning Use and

dissemination of quality approaches in

European e-Learning A study by

the European Quality Observatory”.

<http://libserver.cedefop.europa.eu/veteli

b/eu/pub/cedefop/pan/2006_5162_en.pdf

>.

[9] IEEE. (2000), IEEE-1484.12. 1-2000

Standard for Learning Object Metadata.

[10] IMS. (2003), “Simple Sequencing

Information and Behavior Model-

Version 1.0 Final Specification”.

<http://www.imsglobal.org/simple

sequencing/ssv1p0imsss_

infov1p0.html>.

[11] IMS. (2004), “ IMS Content

Packaging Information Model-Version

1.1.4 Final Specification”.

<http://www.imsglobal.org/content/pack

aging/

cpv1p1p4imscp_infov1p1p4.html>.

[12] ISO/IEC. (1999), ISO/IEC 9126 (1999-

2004) “Software Engineering - Product

quality”. Parts 1-4.

[13] ISO/IEC (2005): 19796-1: Information

technology -- Learning, education and

training – Quality management,

assurance and metrics -- Part 1: General

approach. Geneva: ISO.

[14] ISO/IEC. (2005), Reference Proces

Model of ISO/IEC 19796

<http://www.simbase.co/results/impact-

assessment-model/reference-process-

model-of-isoiec-19796

1/#sthash.Ms9JxSvG.dpuf>. Accessed

March 2014.

[15] Klasnic, K., Lasic-Lazic, J., and

Seljan, S. (2010), “Quality Metrics of

an Integrated e-Learning System -

Students’ Perspective, e-Learning

Experiences and Future”. Safeeullah

Soomro (Ed.), ISBN: 978-953-307-092-

6, InTech.

[16] Massy, J. (2002), “Quality and

e-Learning in Europe Summary report”.

Reading, UK: Bizmedia.

[17] Padayachee, P., Kotze., and van der

Merwe, A. (2010), “ISO 9126

External Systems Quality

Characteristics, Sub-Characteristics And

Domain Specific Criteria For

Evaluating e-Learning Systems”. In:

Moncef Bari and Rachida Djouab

The Eleventh International Conference on eLearning for Knowledge-Based Society, 12-13 December 2014, Thailand

11.7

Proceedings of SACLA Conference.

[18] Parsons, D. and Ryu, H. (2006), “A

framework for assessing the quality of

mobile learning”. 11th International

Conference for Process Improvement,

Research and Education (INSPIRE),

UK, Southampton Solent University,

pp.17-27.

[19] Pawlowski, J.M. (2007), “The Quality

Adaptation Model: Adaptation and

Adoption of the Quality Standard

ISO/IEC 19796-1 for Learning”.

Education, and Training. Educational

Technology & Society, 10 (2), 3-16.

[20] Rekkedal, T. (2006), “Criteria for

evaluating quality in e-Learning (EU

Leonardo project)”.

<http://www.cecoa.pt/Projetos/transncio/

ELQ/Crit%c3%a9rios%20para%20Avali

a%c3%a7%c3%a3o%20da%20Qualidad

e%20do%20e-

Learning.pdf?AspxAutoDetectCookieSu

pport=1.>. Accessed March 2014.

[21] SCORM. (2004), “Shareable Content

Object Reference Model”.

<http://www.scormsoft.com/scorm>.

[22] Watts, H.S. (1989), “Managing the

Software Proces. Chap. 8, 10, 16,

Addison-Wesley, 1989”. ISBN-10:

0201180952 ISBN-13: 9780201180954.

[23] Weibel, S. and Koch, T. (2000), “The

Dublin Core Metadata Initiative-

Mission, Current Activities, and Future

Directions”. D-Lib Magazine, vol. 6,

no. 12.

<http://www.dlib.org/dlib/december00/w

eibel/12weibel.html. Retrieved March

2014>.