Quality in E-learning Ulf-Daniel · 2005-06-09 · Quality in E-learning The learner as a key quality as-surance category Quality will determine the future of e-learning. Many analyses

VOCATIONAL TRAINING NO. 29 EUROPEAN JOURNAL

Cedefop

3

Ulf-DanielEhlersZentrum fürQualitätsforschungim Bildungs- undSozialbereichUniversity ofBielefeld

Quality in E-learningThe learner as a key quality as-surance category

Quality will determine the future of e-learning. Many analyses have reached thisconclusion (1). The KPMG eLearningzwischen Euphorie und Ernüchterung (E-Learning Between Euphoria and Sobri-ety) (2) survey from November 2001 high-lighted the fact that e-learning does notonly depend on good technology, butshould also pay much more attention tothe corporate learning environment andthe learner than it has to date and findeffective ways to incorporate them (3).Berlecon, IDC (International Data Coop-eration) and other institutes are recognis-ing that although e-learning is a growingmarket, it will only develop its full po-tential when we can respond with high-quality e-learning concepts (4).Quality de-velopment and assurance is thereforebecoming a central issue in the e-learn-ing debate (5). Quality assurance conceptsare featuring more prominently. In a widercontext, the growing significance of thequality debate is affecting the entire edu-cation sector, not least as a result of theshock over the results of German pupilsin the PISA study.

Quality development in education, andparticularly in e-learning, is also increas-ingly taking centre stage throughout Eu-rope. The European Commission is cur-rently sponsoring numerous researchprojects for advancing and harmonisingEuropean debate on quality in e-learningas part of a pertinent initiative. For ex-ample, the European Quality Observatory(EQO) project (www.eqo.info) is build-ing an Internet portal with options foranalysing and comparing the different andcompeting quality systems in Europe (6).One of EQO’s main objectives is to pro-mote the integration of quality systems ineducation. Efforts are also underway tosynchronise international debate on qual-ity standards (7).

The particular relevance of quality devel-opment poses two immediate questions:

(1) What is quality? and (2) How can weassure it? Answering these questions is amajor challenge, which we need to faceif we want to raise the future profile of e-learning to the same level as traditionaltraining measures. The good news is thate-learning does not invalidate the basicprinciples of quality assurance. We cantherefore call on tried and tested ideas,models and methods when planning, de-veloping and applying e-learning qualityassurance concepts.

The bad news is that the drawbacks ofconventional approaches remain. Thereinlies the challenge. Ascertaining the actualnature of quality in e-learning opens upan extremely heterogeneous and opaquerealm of concepts, methods and propos-als. First we have to shed light on themystery surrounding the definition ofquality, then we must apply it consistentlyto e-learning.

Quality as a multifacetedconcept

Learning quality – or educational qualityin a wider context – is a multifaceted con-cept. It is not an absolute, it always de-pends on the situation in which it is em-ployed. No European country has reacheda social, political or academic consensuson what educational quality actually is.What we define as quality is therefore anormative setting which refers to a spe-cific context. Consequently, situations andinterests always influence its definition.

This applies even more to social andpedagogical services, since here we can-not follow patterns which are virtual lawsof nature and always turn that which wedeem ‘suitable in its composition to fulfila requirement …’ – i.e. quality (accord-ing to the ISO 900X definition (8)) – intoa negotiable issue between variously pro-pounded academic theories and subjec-

Quality is determining thefuture of e-learning. Manyrecent analyses and devel-opments have reached thisconclusion. Quality assur-ance will therefore acquiregreat significance. Thisposes two questions. Whatis quality? And how can weassure it? Finding answersto these questions is a ma-jor challenge, which weneed to face if we want toraise the future profile of e-learning to the same level astraditional training meas-ures.

The article takes this as astarting point and outlineskey quality assurance fac-tors which put learnerneeds first. It comparessubject-based quality re-search with objective qual-ity criteria. The result is alearner-integrating qualityassurance approach. It ex-amines the latest findingsfrom the largest survey onquality in e-learning to date(Lernqualitaet.de) from anunprecedented learnerperspective.


Cedefop

4

We can distinguish between three funda-mentally diverse elements in the discus-sion:

❏ different interpretations of quality

❏ different players with different perspec-tives of quality

❏ different forms of quality, e.g. input,process, output

Together, these three basic elements pro-vide a frame of reference for all debateon quality.

Quality reflected in various meanings

One dimension of quality is the differinginterpretations of the term. Numerousdefinitions from various fields are avail-able. For example, economics (10) adoptsthe product-based approach, which de-fines quality as a physical characteristic.Thus the quality of a piece of jewellerydepends on its gold content, and the qual-ity of a whiskey on its length of storage.There is also the user-based approach,which relates to individual customer pref-erences. Quality is defined in terms ofuser-friendliness. Under this interpreta-tion, optimal fulfilment of demand signi-fies the best-possible quality. An oft-bor-rowed book therefore attains a higherquality than a seldom-loaned volume.User preferences are the determinants. Wealso have the production-based approach,which stems from the manufacturer andsets standards, compliance with whichequals quality. Here, the primary goal isfunctionality, so a Swiss precision watchcould have the same quality as a no-nameproduct from Hong Kong. All books thatdo not fall apart have the same quality.

Of course, these attempts at definition can-not simply be transferred to the educationsector. Unlike business, education does notinvolve classic supplier-customer relation-ships. It is an association of co-producers.An e-learning programme supplies tech-nology and content, but it is down to learn-ers themselves to actively use it, i.e. learn.The interaction between the learning pro-gramme and the learner is known as a co-producer relationship (11).

In education we can currently identifyaround five different meanings or inten-

QDifferent perspectives/players

Different quality interpretations

Differentqualities

Figure 1:

Quality dimensions

tive political and social interests. Donabe-dian (1968) highlights the relational char-acter of the term in his definition of qual-ity. ‘Quality is the degree of conformancebetween a performed service and thegoals set for the service (9)’. In this sense,quality describes a relationship. Evidently,discussion of quality involves consider-ing a metatheoretical category which con-cerns the composition of the object un-der scrutiny.

Figure 1:The origins and impact of the quality de-bate, which now encompasses a largesection of society and affects many socialsub-segments, are no longer easily vis-ible. However, we can observe that qual-ity is becoming an increasingly importantand desirable category in individual andsocial contexts. We can regard qualitymore and more as a subjectively individualand collectively influential category.Achieving good/top quality is a hotly de-bated and much-sought-after goal in allsectors of society. Even language bearswitness to the concept’s significance, asthe renaissance of phrases such as ‘qual-ity of life’, ‘quality of services’, ‘productquality’ and ‘water quality’ shows. Thevery impact of the word ‘quality’ on be-haviour demonstrates its meaning. Theword merely signifies ‘composition’ (Latinqualis), but in everyday language it isused to distinguish a characteristic of anobject as being of a higher calibre thanthat of another object.

To critically analyse quality, it is helpfulto identify the basic points of the debate.

(1) See also Ehlers et al., 2003, par-ticularly Chapter 6.

(2) Cf. KPMG, 2001.

(3) A UnicMind study, ‘E-Learning andKnowledge Management in MajorGerman Enterprises’, also airs thisproblem (2001).

(4) Berlecon Research (2001) elabo-rates this subject in its study Wachs-tumsmarkt E-Learning: Anforde-rungen, Akteure und Perspektiven imdeutschen Markt (E-Learning GrowthMarket: Challenges, Players and Pros-pects in the German Market).

(5) See also Ehlers, 2002 on develop-ment of e-learning.

(6) Pawlowski (2003) describes theresearch project in detail.

(7) For example, Germany’s DeutschesInstitut für Normung (DIN), Europe’sCEN and ISSS and the ISO organiseworkshops on this topic.

(8) We can find an explanation of theISO quality definition and parametersin Gräber 1996, for example.

(9) Donabedian, 1968.

(10) Cf. Müller Böling, 1995.

(11) Cf. Fendt, 2000, p. 69; Ehlers 2003,Chapter 3, for example.


Cedefop

5

tions for the term ‘quality’ (12), some ofwhich resemble the definition examplesin economics:

❏ quality as an exception describing thesurpassing of standards

❏ quality as perfection describing thestate of flawlessness

❏ quality as functionality referring to thedegree of utility

❏ quality as an adequate return measuredby the price-performance or cost-benefitratio

❏ quality as a transformation describingthe above-mentioned co-producer rela-tionship between the learner and thelearning programme and referring to thelearner’s progress via a learning process.

Quality in the clash of perspectives

However, there are not only different in-terpretations of quality but also variousplayers’ interests and perspectives (13): theenterprise – the user of the training meas-ure, the tutors supervising an e-learningprogramme, the personnel managers whoestablish the framework for continuingtraining in their sector, and the learner.Each of these four players generally hasdivergent interests and differing qualityrequirements and interpretations. We musttherefore regard quality not as a staticelement, but as a negotiable relationshipbetween the players involved in the so-cial process.

Quality at all levels

Last but not least, quality can also referto different education levels or processes.We can cite the different levels of the fa-mous quality triad (according to Donabe-dian) as examples:

❏ training measure prerequisites (input/structure quality): availability and capa-bility of computers or tutor qualificationfor of e-learning;

❏ the learning process (process qual-ity), i.e. the interaction of learners, learn-ing format, corporate learning culture,learning content and desired training goals– or;

❏ the result (outcome quality) of e-learning, i.e. the increase in learners’ pro-fessional competence.

Defining quality therefore involves navi-gating this multidimensional space (14).There is no easy answer or standard qual-ity assurance solution. We have (unfortu-nately) had to abandon the hope of onlyhaving to define quality criteria once tobe able to appraise e-learning courses andformats properly in future. Ultimately, e-learning measures strive to boost learn-ers’ professional competence. A key fac-tor in e-learning will thus be quality ori-entation which spans all processes andputs learners first. They must take poleposition in e-learning quality concepts,since their professional competence is atstake (15). Researchers and politicians arecalled to establish exactly what this in-volves. However, we can already clearlyidentify several fundamental requirementswhich the new learning formats and so-cial developments impose.

Learner-integrating qualityassurance

It is imperative that we do not perceive anew focus on learner needs merely as awhim of fashion. Below we will there-fore list and explain basic dimensions oflearner-integrating quality assurance. Wecan use two main arguments to justifystrengthening the learner perspective:

❏ a paradigm shift with consequencesfor defining learning quality is appropri-ate and necessary for quality assurancein e-learning;

❏ investigating quality from a user per-spective entails integrating learner-ori-ented quality development at all levelsof the training process.

The following section lists reasons forprominently integrating learner variablesin quality models and traces the conse-quences for the quality assurance proc-ess.

Paradigm shift in quality assurance

A paradigm shift has been evident in con-tinuing vocational training for some time:a move from lecturing to enabling – from

(12) Harvey/Green, 1993, p. 9ff presentan analysis of interpretations of qual-ity in education. Wallmüller, 1990, p.7f also features a systematic assess-ment of quality concepts, though thisis more generalised and is not spe-cifically related to education.

(13) Cf. Fischer-Bluhm, 2000, p. 680;Ehlers, 2003, Chapter 3.

(14) Cf. Fischer-Bluhm, 2000, p. 680f.

(15) Professional competence here hasa broader sense of ‘ability to interactcreatively with one’s environment ina complex world’.


Cedefop

6

Learning variablesprevious knowledge,

attitudes, experience

Learner goal/intention/transfer goal

knowledge, professional competence acceptance, motivation

Learning topic/learning material

Learning format

(1)

(2)

Learning context: learning centre,class group, place of work, corporate learning cultureMultimedia learning format/learning software

QQQQQ

Instructional condition

Instructional condition

Instructional methods

Instructional outcomes

behaviourist to cognitive didactics andfrom instructionalist to constructivist ap-proaches (16). E-learning opportunities aregenerally radicalising this development.They pave the way for previously non-existent need orientation and programmeindividualisation – not least by replacingteaching along the lines of ‘one for many,now and here (Taylorist principle) withteaching/learning organisation which ad-vocates ‘need-oriented learning, any time,anywhere’.

On the whole we can see that learnersare being granted more and more say indefining continuing training quality (17).This applies to e-learning in particular.We can identify four reasons for this de-velopment: (a) from an economic pointof view, learners are having to make anincreasing financial contribution, eitherdirectly, by paying for private furthertraining measures, or indirectly, by sacri-ficing leisure time to participate in fur-ther or continuing training organised bythe company; (b) from an educationaland didactic point of view, this is re-flected in less lecturing and more hands-on experience. This change originates in

a shift from behaviourist learning theo-ries to cognitive learning models and frominstructionalist to constructivist ap-proaches. This generally involves lessstandardisation and more orientation tosituation and subject; (c) at a social level,one reason lies in the emergence of theknowledge society. Globally availableknowledge currently doubles every fourto five years. U.S. sociologist RichardSennet (18) predicts that American collegestudents will change jobs eleven timesduring the course of their career and com-pletely renew the basis of their knowl-edge three times. This lifelong learningprocess is individual and cannot be stand-ardised. The challenge for e-learning pro-grammes is to match individual require-ments as closely as possible to the avail-able learning formats; (d) e-learning’sspecial traits are a fourth reason forlearners’ increasing influence on defini-tion of quality in this area (19):

Access and learning forms: individualaccess to software has eliminated the needfor standardised times and shared, publiclocations for learning processes. Learningprimarily takes the form of private study,

(16) Harel et al., 1999; Jonassen, 1996;Reeves, 1999; Wilson et al., 2001 andothers describe this development.

(17) Cf. Gnahs, 1995.

(18) Cf. Sennet, 1998.

(19) Cf. Baumgartner, 1997.

Figure 2:


Cedefop

7

often at home, independently of teachersor other learners. E-learning facilitates in-dividual, unrestricted access to material anytime and anywhere. It also offers a wealthof learning forms: classroom phases, vir-tual self-study, tutor support, working andsharing with other learners. This range ofpossibilities allows individual tailoring oflearning opportunities.

Status quo and needs structure: groupsof e-learners may have a heterogeneousstatus quo. This applies particularly todemographic components (professionalstatus, level of education, etc.), content(previous knowledge, skills, etc.) andlearning experience. Uniform learningbackgrounds within the group, such asthose found in a classroom-based scenario,cannot automatically be assumed. This hasa special bearing on the option of tailor-ing learning processes to individuals.Whereas traditional group teaching onlyworks with relatively homogeneous back-grounds, e-learning is not subject to thisrestriction. Moreover, learners themselvesare responsible for determining when, howlong and how often they study.

The need structure usually has a simi-larly heterogeneous status quo. The en-visaged objectives and related motivationstructures of media-supported learning areentirely open. However, these diverseneeds not only influence learning success,but are also instrumental in defining it.The number of goal structures is mani-fold: course participation inspired by per-sonal interest in a subject is just as feasi-ble as acquisition of an additional pro-fessional qualification or continuing train-ing to gain a skill which has become in-dispensable for normal working life. Agroup of learners on one e-learning pro-gramme may pursue a very disparaterange of goals.

Freedom of programme selection: e-learning gives learners the chance to de-cide for themselves how quickly they wishto learn, at which times and which sec-tion they want to focus on, skip or inten-sify. Learners can largely choose their owntopics, schedules and strategies. Eachperson can decide individually whichcontent, section or course they digest,when and how. Scope for such individualorganisation of learning also engendersindividual quality requirements.

To summarise, all four reasons – eco-nomic, educational/didactic, social andspecial e-learning traits confirm that learn-ers should be more instrumental in de-fining the nature of learning quality. Forquality assurance this specifically meansthat learner variables gain more weightin relation to the other learning processvariables.

The ‘paradigm for constructing and evalu-ating multimedia learning environments’,which Rainer Fricke (20) devised, drawingon proposals from Reigeluth (21) and him-self (22), clearly illustrates the relationshipbetween learner variables and learningformats. Fricke avers that a learning for-mat’s effectiveness depends on four fac-tors:

❏ the learning environment: both themultimedia system (LMS) and the sociallearning environment (corporate learningculture, locations, etc.);

❏ the learner or the learner variables:previous knowledge, education, learningskills, media skills, interests, expectations,goals, etc.;

❏ the learning topic: content and pres-entation of study material;

❏ the (intended) learning outcome orobjective: goals to be achieved via learn-ing, such as promotion as a result of in-creased professional competence.

An e-learning format only takes shapewhen all four of the listed factors unite.All four aspects influence learning suc-cess, learning effectiveness and quality.The quality of an e-learning format there-fore depends on the interaction of all fourfactors. The depicted paradigm shiftboosts the significance of the learner vari-ables within the entire learning format.This does not mean disregarding all otherfactors and treating learner needs thesame as before but using technologicalor content-based/curricular learning proc-ess aspects. It merely involves shiftingthe emphasis on learner factors.

Four consequences for quality assur-ance in e-learning

Giving learners a key position in qualityassurance has consequences for what we

(20) Cf. Fricke, 1995, p. 405.

(21) Cf. Reigeluth, 1983, p. 22.

(22) Cf. Fricke, 1991, p. 15.


Cedefop

8

define as quality in e-learning. The fol-lowing four points will explain their rel-evance for quality assurance in e-learn-ing programmes.

Consequence 1: From technologicalorientation to clear user orientation

The maxim for e-learning should be ‘tech-nology-supported but not technology-ori-ented’. The quality of the course does nothinge on what technology can achieve,but on how the appropriate technologycan be used to tailor content, methodsand situation to learners’ needs. Studiesshow that (qualitatively) poor CBT whichis sensibly embedded in a learning for-mat can produce better results than (os-tensibly) good programs which are notused sensibly (23).

Consequence 2: From programme ori-entation to learner orientation

This does not just mean rehashing thegenerally accepted principle of participantorientation without further reflection andmerely considering how programme pres-entation and design can be made moreuser-friendly and adapted to the specificsituation. This is undoubtedly important

and correct. However, the paradigmaticshift goes one step further. It involves achange in perspective. Hence the ques-tion is not what training measures can beoffered to staff, but what do they neednow and in the future in which areas, andwhat forms of teaching suit their careerand level of education?

Consequence 3: Quality originates inlearning (and with the learner)

The depicted paradigm shift has far-reach-ing consequences for defining quality andquality assurance in e-learning. This view-point does not regard learners as courseconsumers but as co-producers of theirown learning success. From this perspec-tive the widely advocated economic viewof the learner as a customer is unaccept-able. On the whole it appears that the com-mon definition of quality as ‘what the con-sumer wants’, now popular even in edu-cation, which was triggered by the trans-fer of different quality management phi-losophy principles (e.g. TQM) from indus-try to social services, and to educationsince the late 1980s (Berwick, 1989), shedslittle light on what quality really is. It isnot even easy to establish who the con-sumer in vocational training is. Is it the

QF5: Information transparency of course/provider

QF6: Course structure/classroom sessions

Focus on interactionLearning supervisionLearner vs. content orientationIndividualised leaner supportPromotion of development and learningTrad./synch./asynch. communication media

Background information to course material Multimedia preparation of materialDesign and structure of materialLearning supportTests/exercisesIndividualised task setting

Personnel courses and learning process supportIntroduction to technology and contentExaminations

AdviceInterdisciplinary informationInformation on course content

Individualisation and need orientationIndividual non-economic costsEconomic costsTransfer expectationsExternal interest in advantages

Adaptive operation and personalisationSynchronised communication optionsElectronic availability of content

Social cooperation dimensionDiscursive, virtual cooperation dimension

QF1: Tutor supportQF2: Communication and cooperation

QF3: Technology

QF4: Costs - expectations - benefits

QF7: Didactics

(23) Cf. Schenkel, 1995, p. 22.

(24) Cf. Müller Böling, 1995.

(25) Cf. Behrendt, 1998.

(26) Cf. Meier in Schenkel, 1995;Zimmer/Psaralidis, 2000; Behrendt,1998; Schenkel, 1995; Behrendt, 1998,p. 43ff; Schenkel, 1995, p. 13ff.

Figure 3:


Cedefop

9

organisation making a demand – the com-pany which sends its staff on further train-ing – or is it the course participants them-selves? Moreover, learning processes arenot a service which an education providermust supply to or for a potential customer.They require the cooperation of the ‘cus-tomer’ (i.e. learner). We therefore assumeco-production of the training product ineducation and social affairs.

Quality only results when the learner in-teracts with the learning format: i.e. onlywhen learning takes place (co-productionin learning success). An e-learning for-mat has no learning quality in itself. Itmerely supplies the framework (the for-mat) to support the learning process.

Consequence 4: Quality promotioninstead of quality assurance?

Most of the evaluation concepts for as-sessing learning software applied in qual-ity assurance processes today follow a so-called mechanical model of impact re-search (Müller-Böling also refers to an exante evaluation model (24)). Learning soft-ware which has been tested according tocertain quality criteria (e.g. AKAB, MEDA’97, etc.) is used to achieve a particulareffect – growth in workers’ professionalcompetence. This neglects to consider thatevaluation of learning software says noth-ing about its potential impact. Rather, in-vestigations such as the case studies ErichBehrendt has performed in several sec-tors (25) reveal that not only the learningsoftware but also the learning formats, thecorporate working, learning and manage-ment cultures and learner motivation, ac-tivity and guidelines for behaviour aredeciding factors for effective e-learning.This confirms that learning quality in-volves the interaction of various givens,only one of which is software.

Quality only ensues from the harmonisa-tion of learning requirements, the e-learn-ing format and other contextual aspects.Prognostic standards barely hold water asforecasts of learning quality (26). After all,quality can only be achieved through con-stant optimisation – or promotion – of thisinteraction. It therefore makes sense tostrive for quality promotion rather thanquality assurance in future, since onlystandards can be assured. That does notmean that from now on, all standards

Figure 3:

Subjective quality model

Quality segment 1: Form of tutor supportQuality requirements which learners regard as significant for online tutor support:assigned tasks, behavioural guidelines, rights to tutor time, tutor qualification re-quirements and information on desired communication media.

Quality segment 2: Online course cooperation and communicationQuality requirements which learners impose on communicative and cooperativeonline learning processes. These concern specific forms and options of communi-cation between the learning process participants, and their design (28).

Quality segment 3: Learning technologiesQuality requirements which learners impose on selected technological compo-nents in an online learning platform.

Quality segment 4: Costs – expectations – benefitsLearners’ cost-benefit considerations also play a vital role in e-learning qualityassessment. Learners’ investment in online continuing training and the ensuingresults, e.g. improved professional competence in a work context, must be in anappropriate ratio (29).

Quality segment 5: Information transparencyQuality requirements for information on online training providers and the coursesthey offer. The main question is: What course and provider information shouldlearners have access to, and what advice services are needed to create transpar-ency?

Quality segment 6: Classroom sessionsQuality requirements for course structure, particularly classroom sessions: designand frequency of classroom sessions, study advice services, timetables and organi-sation, evaluation of online courses.

Quality segment 7: DidacticsThe didactics segment covers content, learning goals, methods and materials. Qualityrequirements chiefly affect the following aspects: course material background in-formation, media-compatible multimedia material design, sectioned and structuredcourse material, promotion of learning competence, feedback via exercises andmonitoring of learning progress, individual tasks adapted to learners’ personalgoals and abilities.

(28) These can be learner-learner interaction, learner-tutor interaction, learning group-tutor interac-tion, learner-expert interaction, etc. The element of learner cooperation is seen to be linked veryclosely to the concept of collaboration, i.e. cooperation here does not signify work division, butdescribes shared cognition in a spatially divided learning system.

(29) However, costs are not restricted to the financial aspect. The time learners invest, the difficulties ofself-organisation and commitment to self-motivation involve considerable sacrifices for learners. Onthe plus side, learners can harvest various other fruits from their efforts as well as a financial ROI.

should be annulled, but it emphasises thatproviders and services are only the frame-work model. Learners are the primarysculptors of their e-learning process andhence the learning quality. It is they who


Cedefop

10

QLernqualitaet.de – quality researchfrom a learner perspective

The lernqualitaet.de – Qualität ausLernersicht (Quality From a Learner Per-spective) study (27) takes this issue as itsstarting point. It has two innovative goals.First, it aims to define quality from alearner point of view. Second, it movesaway from the common practice of ap-plying the same quality equally to alllearners. Instead, it creates target-group-specific quality profiles. The results showthat quality criteria do not apply equallyto all learners. These findings prove thatthe common practice of developing qual-ity according to generally applicable cri-teria makes no sense.

The research project provides the first-ever empirical classification of subjectivee-learning quality requirements. This nowsets the stage for viable and appropriatequality development for networked learn-ing courses focusing on learner needs.

Comprehensive e-learning servicesfor learners

A subjective quality concept for onlinelearning formats must consider more andbroader determinants than ‘mere’ didac-tic or technological aspects, as the sur-vey results show. They reveal that learnerquality requirements for e-learning can bedivided into seven quality segments:

It is clear that learners in a self-directedlearning process desire assistance from anetwork of supporting learning services.Moreover, we can see that learners’ qual-ity requirements apply to the entire con-tinuing training process and not just thelearning process. Learners also attachimportance to learning conditions (struc-ture quality) and results (output quality).

Target-group-oriented quality con-cepts

The survey shows that quality require-ments differ from learner to learner anddepend on various aspects, such as edu-cational experience, individual learningcompetence and socioeconomic factors.The survey compiles target-oriented qual-ity profiles from the wealth of individualquality requirements, which appropriately

Figure 4:

Target-group-specific quality profiles(selected characteristics)

The results-orientedlearner(N=235)Independent and goal-oriented• Individualisation• Standard programmes• Work-integrated learning• Instrumental purpose orientation• Learning and media competence• Classroom sessions, interaction and

communication

The avantgardist

(N=392)Interaction-oriented• Discussion/communication• Learner-oriented tutor support• Avant-gardist media/tehnology• Virtual learning groups• Information & advice• Variety of techning methods

The individualist

(N=328)Content-oriented• Content-based quality requirements• Individualised programmes• Didactic structure• Self-directed leqarning• Classroom sessions, interaction and

communication

The pragmatist

(N=293)Need-oriented• Individualised programmes• Practical tutor support• Non-economic costs• Information & advice• Personalisation of the LP• Didactic requirements

learn. It goes without saying that e-learn-ing courses must conform to minimumstandards. But minimum standards do notthemselves imply any learning quality.

Learner-oriented qualitydevelopment

The vital role of the learner, which thefour cited consequences for quality as-surance in e-learning stress, indicates thefuture need for reliable learner-orientedquality research. This particularly involvessetting determinants for quality definitionfrom a learner point of view and fixingthem so that they can be employed inconstructing concrete e-learning pro-grammes. The fundamental question istherefore, what are significant factors ofe-learning quality from a learner perspec-tive? The status of learners must be re-considered.

(27) The study was performed at theUniversity of Bielefeld (Ehlers 2003)and will be published in Germany inspring next year (2004). More infor-mation on the Internet at http://www.lernqualitaet.de.


Cedefop

11

reflect the existing differences in e-learn-ers’ quality demands. These prototypicalprofiles can act as models for learner-ori-ented quality development. Analysis sug-gests that the following target group struc-tures would be viable (Figure 4):

❏ Content-oriented individualists,who want to learn independently and donot consider tutor support vital for highquality. They have little need of supportservices. Their quality requirements onlyconcern content. They attach no impor-tance to communication and interaction.

❏ Independent goal-oriented or re-sults-oriented learners, who use e-learning chiefly to achieve a previouslydefined objective and require only thesupport necessary for this. They are sat-isfied with standard programmes and at-tach no importance to individually tailoredlearning formats.

❏ Need-oriented pragmatists are inter-ested in communicative discussion butadopt a practical approach, focusing onwhat is absolutely necessary. Extreme in-dividualisation of the learning programmeis not a priority, and they do not regardspecial application of various media as aprerequisite for high-quality learning for-mats.

❏ Interaction-oriented avant-gardistsperceive quality in a comprehensive pal-ette of support services. They are inter-ested in improving learning competenceas well as attaining specific goals. Theyrequire an interaction-oriented learningformat which incorporates a rich and var-ied use of media.

Figure 4:We can assign a different set of qualityfactors from the seven subjective qualitymodel segments to each target group.The study proves that quality develop-ment for learning formats intended tofacilitate self-directed learning must takevery different learner quality definitionsand requirements into account. This ap-plies particularly to guidance serviceswhich take the form of advice or tutorsupport. Consideration of target-group-specific quality needs is therefore a stra-tegic success factor for providers of con-tinuing e-training. If e-learning is to reapthe benefits of these new findings, it mustdevelop tools which integrate them intoall levels of the electronic continuingtraining process. In the next section wewill present a feasible evaluation ap-proach which analyses the different edu-cation process stages, apply it consist-ently to e-learning and couple it with thelearner orientation model developedabove.

Figure 5:

Evaluation model for comprehensive quality assurance (Kirkpatrick, 1994)

Evaluationin the planningstage

Evaluationin thedevelopmentstage

Evaluationin theimplemen-tation stage

Evaluationof impact

reactionlevel

learninglevel

actionlevel

successlevel


Cedefop

12

Evaluation in the planning stage

Evaluands❏ Target-group-specific prerequisites: Previous knowledge, interest in the quality measure, expectations, partici-

pant media application skills, information on educational context, learning competence, self-organisation/self-di-rection, how does the learner define learning success? (subjective quality model and target-group-related qualityconcepts)

❏ Contextual prerequisites: onsite organisational/technical requirements, corporate learning culture, private learn-ing environment, etc.

Methods❏ Qualitative methods: survey, observation

Evaluation in the development stage

Evaluands❏ Target-group-applicable pedagogical design of course/learning modules and course structure (e.g. blended learn-

ing vs. purely virtual self-study)❏ Ergonomics❏ Acceptance❏ Motivation effects❏ Do the concepts suit learner requirements? (subjective quality model and target-group-related quality concepts)

Methods❏ Formative evaluation through iterative optimisation loops, observation, behaviour recording, log file analyses, tests

and checklists

Evaluation in the implementation stage

Evaluands (subjective quality model and target-group-related quality concepts)❏ Reaction level: How do learners react to the course? What form should the e-learning programme take to generate

a positive reaction from learners?❏ Learning level: What are the learning results? How can the learning process be influenced positively?❏ Activity level: Were learners able to use the learning content for their own purposes, e.g. transfer it to their work

situation? Has professional competence increased (reconstructive analyses)? What form should an e-learning coursetake to support learners in developing initiative and competence?

❏ Level of success: What impact does continuing training have on learners’ situations, e.g. their status in the com-pany or workplace? Was the e-learning measure a success for the learner?

Methods❏ Combination of qualitative and quantitative methods: Surveys, observations, tests

Evaluation of impact

Evaluands❏ Has the continuing training measure achieved the desired success (e.g. in a corporate/organisational context)?❏ Have the individual or organisational goals (corporate training) been attained?❏ Is the desired increase in the learner’s professional competence visible?

Methods❏ Cost calculation (ROI), staff discussions (quality management), reconstructive analyses of increase in professional

competence


Cedefop

13

Assuring quality at all levels

Until recently, monitoring the success ofcontinuing training measures was limitedto asking participants how satisfied theywere, and at best, to examining directlearning outcomes (30). Now, however, de-tailed evaluation models are also avail-able for e-learning (for learning softwareand online learning formats). In generalwe can affirm that quality assurance ismore than simply monitoring direct learn-ing effects. It considers all determinants(see Figure 5): learner, learning topic,desired results and both the technologi-cal and the social learning environment(place of work, corporate learning cul-ture, private learning situation, etc.).

Quality assurance does not solely consistof good planning and preparation. It mustextend to all phases of the training proc-ess and involve the learner. It begins withquality awareness and ends with assur-ance of the transfer function, which is theultimate aim. A tailored evaluation con-cept is also essential for ensuring quality.

It is vital to consider every stage of thetraining process, from planning, throughdevelopment and implementation, rightup to transfer assurance.

Figure 5:It is a widespread misconception that e-learning has only to be well planned andprepared to achieve the desired outcome.A detailed quality assurance conceptmight mirror Figure 5. It spans all stagesof a qualification process and goes be-yond standard evaluation procedures (31)for continuing training measures by in-corporating the planning and develop-ment stage and the impact of the meas-ure (e.g. return on investment in companytraining) as well as assessing the imple-mentation stage in detail.

Each level must consider the four fac-tors of learning format, learner, learningcontent and learning outcome/intention.Consideration of learner variables is par-ticularly important. Thus, quality assur-ance involves integrating the learner inevery phase of planning, developmentand execution. Learner-integratingquality assurance should therefore in-clude at least the following evaluationquestions, which render it possible (sum-marised on page 12).

Tables

Figure 6:

Media competence model according to Baacke

Media know-how Media use

Media criticism Media creativity( )Figure 7:

Assuring quality means actively strengthening learn-ers’ learning competence

Media competence• Media use• Media know-how• Media criticism• Media creativity

Self-directed learning• Needs, goal• Materials, contents• Methodological compet.• Learning opportunities

Outlook: Promotion oflearning competenceas a quality condition

It is clear that learners are becoming morerelevant and taking centre stage in qual-ity development. However, a further as-pect is essential for establishing qualityin learning: the learning competence oflearners themselves. In 2001 StiftungWarentest (the German consumer evalu-ation foundation) published a surveywhich examines whether a StiftungBildungstest (education evaluation foun-dation) would be advisable in the Ger-man training provider landscape. It con-cluded that an institution of this kind isurgently needed, because …

‘… autonomous decisions by private us-ers presuppose the existence of comprehen-sible markets. Ideally consumers shouldknow what is on the market, in whatqualities the desired product is availableon the market and what price-perform-ance ratio is appropriate to their require-ments.’ (Stiftung Warentest Dec. 2001)

(30) Bliesener, 1997, p. 163-167.

(31) Kirkpatrick proposes a four-stageevaluation model for training pro-grammes, which still constitutes thestandard evaluation approach in theU.S. (Kirkpatrick, 1994).


Cedefop

14

However, to make these decisions, learn-ers must first identify their needs. Theymust know what training they require orwish to obtain in the course of their life-long learning process. They must developtheir own definition of quality. They mustbecome their own training manager.

Only when learners are capable of doingthis will they be in a position to proactivelyshape their own learning success and posi-tively influence learning quality. E-learn-ing providers and enterprises/other workorganisations are ‘only’ facilitators in thisscenario. Learners can then assume a highdegree of responsibility for the learningquality of training processes. For bothlearners and providers bear responsibilityfor the learning process and hence for thequality of all learning in the co-producerrelationship between training programmeand learner. Assuring quality therefore al-ways entails strengthening learner compe-tence in this way. Against this backdrop,‘training for self-training’ (learning to learn)gains significance. What is necessary foracquiring this competence?

Training programmes should aim to im-part media skills to learners. They mustalso foster media creativity and a criticalapproach to media-based informationover and above pure media competence.According to Baacke, comprehensivemedia competence comprises four dimen-sions (32):

Figure 6:

The ability to acquire knowledge via themedia, use it creatively and reflect criti-cally on media-imparted communicationprocesses and information is vital for ef-fective media-based learning.

Fig. 7:Supporting and promoting self-directedlearning proficiency continues to be vi-tal. Learners should be equipped to ana-lyse their own education needs, for-mulate goals and access and exploitmaterials and content independently.To do this they require a high level ofmethodological competence and thecapacity to arrange their own learningopportunities. It is not merely a caseof empowering learners to solve indi-vidual, isolated problems. It also in-volves comprehensive training in a rap-idly changing social and professionalworld. Otto Peters (33) emphasised thisconnection.

‘The question of whether we … advocateself-directed learning is no banality, …in view of the social and cultural prob-lems we are likely to encounter it is asheer survival tactic.’

Above all, assuring quality in e-learningalso involves assuring learning compe-tence in learners. This enables them toshape high-quality learning processesthemselves and to make the most of pro-gramme potential within the learning for-mat.

Baacke, D. Medienkompetenz. Tübingen: 1997.

Baumgartner, P. Evaluation vernetzten Lernens: 4Thesen. In: Simon, H. (ed.). Virtueller Campus. For-schung und Entwicklung für neues Lehren und Ler-nen. Münster: 1997, p. 131-146.

Behrendt, E. Multimediale Lernarrangements imBetrieb. Grundlagen zur praktischen Gestaltungneuer Qualifizierungsstrategien. Bielefeld: 1998.

Berlecon Research Wachstumsmarkt E-Learning:Anforderungen, Akteure und Perspektiven im deut-schen Markt. Berlin: 2001 Available on the Internet:h t tp ://www.ber lecon.de/s tudien/e learn ing/index.html

Bliesener, T. Evaluation betrieblicher Weiterbildung.In: Schwuchow, K.; Gutmann, J. (eds.). Weiterbil-dung Jahrbuch 1997. Düsseldorf: 1997, p. 163-167.

Ehlers, U. E-Learning: Stand und Perspektiven.Survey for Verbraucherzentrale Bundesverband e.V.,Bielefeld: 2002.

Ehlers, U. et al. (eds.). E-Learning-Services imSpannungsfeld von Pädagogik, Ökonomie und Tech-nologie. L_-Lebenslanges Lernen im Bildungs-netzwerk der Zukunft, Bielefeld, 2003.

Ehlers, U. Qualität beim E-Learning. EmpirischeGrundlegung und Modellkonzeption einer subjekt-orientierten Qualitätsentwicklung. Dissertation, Bie-lefeld, 2003.

Fend, H. Qualität und Qualitätssicherung im Bil-dungswesen: Wohlfahrtsstaatliche Modelle undMarktmodelle. Zeitschrift für Pädagogik No 41. Sup-plement (2000): Qualität und Qualitätssicherungim Bildungsbereich: Schule, Sozialpädagogik, Hoch-schule. Published by Helmke, A., Hornstein, W. andTerhart, E.. Weinheim und Basel.

Fischer-Bluhm, K. Qualitätsentwicklung als Ant-wort auf die Individualisierung im Bildungsbereich– am Beispiel der Hochschulen. Arbeitsstab ForumBildung (ed.): first congress of the Forum Bildungon 14 and 15 July 2000 in Berlin. Bonn: 2000, p.680-693.

Bibliography

(32) Baacke states that the concept ofmedia competence is based on theconcept of communicative compe-tence. He presented this as theBielefeld media competence model inBaacke, 1997.

(33) Cf. Peters, 1999.


Cedefop

15

Fricke, R. Zur Effektivität computer- und video-unterstützter Lernprogramme. In: Jäger, R.S. et al.(eds.). Computerunterstütztes Lernen (Supplement2 to Zeitschrift Empirische Pädagogik). Landau:1991, p. 167-204.

Fricke, R. Evaluation von Multimedia. In: Issing, L.J.; Klimsa, P. (eds.). Information und Lernen mitMultimedia,. Weinheim: 1995, p. 400-413.

Gnahs, D. Zwischenbilanz der Qualitätsdebatte. In:Deutsches Institut für Erwachsenenbildung (DIE)et al. (eds.). Literatur- und Forschungsreport Wei-terbildung June 1999. Report 43. Topic: Quality.Wissenschaftliche Halbjahresschrift des DIE. June1999. Bielefeld: 1999.

Gräber, W. Kriterien und Verfahren zur Sicherungder Qualität von Lernsoftware in der beruflichenWeiterbildung. Report commissioned by the FederalInstitute of Vocational Training. Kiel: 1996.

Harel, I.; Papert, S. (eds.). Constructivism .Norwood, New Jersey: 1999.

Harvey, L.; Green, D. Qualität definieren. Fünf un-terschiedliche Ansätze. Zeitschrift für Pädagogik No41. Supplement (2000): Qualität und Qualitätssi-cherung im Bildungsbereich: Schule, Sozialpäd-agogik, Hochschule. Published by Helmke, A., Horn-stein, W. and Terhart, E.. Weinheim und Basel

Herzberg, F. The Motivation To Work. New York:1967.Jonassen, D. (ed.). Handbook of EducationalCommunications and Technology. New York: 1996.

Kirkpatrick, D.L. Evaluating Training Programs.The Four Levels. San Francisco: 1994.

KPMG: eLearning zwischen Euphorie und Ernüch-terung. Munich: KPMG: 2001. Available on theInternet: http://www.kpmg.de/about/press-office/2001/11/28.html

Meier, A. Qualitätsbeurteilung von Lernsoftwaredurch Kriterienkataloge. In: Schenkel, P.;Holz, H.(eds.). Evaluation multimedialer Lernprogrammeund Lernkonzepte. Berichte aus der Berufsbildungs-praxis. Nuremberg: 1995.

Müller-Böling, D. Qualitätsmanagement in Hoch-schulen. In: Lohse, H. (ed.). 6. Deutscher Biblio–thekskongress. 84. Deutscher Bibliothekartag inDortmund 1994: Arbeitsfeld Bibliothek. Frankfurt/M: 1994, p. 75-83.

Müller-Böling, D. Qualitätssicherung in Hochschu-len, Grundlage einer wissenschaftsbasierten Gesell-

schaft. In: Müller-Böling, Detlef (ed.). Qualitätssi-cherung in Hochschulen. Forschung, Lehre, Mana-gement. Gütersloh: 1995, p. 27-45.

Pawlowski, J . M. The European Quali tyObservatory (EQO): Structuring Quality Approachesfor E-Learning, ICALT 2003, Athens, July 2003.

Peters, O. Auf dem Weg zum selbstgesteuerten Ler-nen. Hagen: 1999. Available on the Internet: http://www.fernuni-hagen.de/ZIFF/kurslist.htm

Reeves, T. A Research Agenda for InteractiveLearning in the New Millennium. Proceedings: Ed-Media ’99. Seattle, WA. AACE. Charlottesville, VA:1999.

Reigeluth, Ch. M. Instructional Design: What Is Itand Why Is I t ? In : Reigeluth , C. M. (Ed.) .Instructional Theories and Models: An Overview ofTheir Current Status. Hillsdale, New Jersey: 1983,p. 3-36.

Schenkel, P. Einführung. In: Schenkel, P.; Holz, H.(eds.). Evaluation multimedialer Lernprogrammeund Lernkonzepte. Berichte aus der Berufsbildungs-praxis. Nuremberg: 1995, p. 22.

Schenkel, P. (ed.) Qualitätsbeurteilung multime-dialer Lern- und Informationssysteme. Evaluations-methoden auf dem Prüfstand. Nuremberg: 2000.

Sennet, R. Der flexible Mensch. Die Kultur des neu-en Kapitalismus. Berlin: 1998.

UnicMind. eLearning und Wissensmanagement indeutschen Großunternehmen. Göttingen: 2001.Available on the Internet: http://www.unicmind.de/elearningstudie.pdf

Wallmüller, E. Software Qualitätssicherung in derPraxis. Munich: 1990.

Wilson, B.; Lowry, M. Constructivist Learning onthe Web. In: Burge, L. (ed.) Learning Technologies:Reflective and Strategic Thinking. Jossey-Bass. SanFrancisco: 2001. Available on the Internet: http://ceo.cudenver.edu/~brent_Wilson/WebLearning.html

Zimmer, G.; Psaralidis, E. Der Lernerfolg bestimmtdie Qualität einer Lernsoftware! Evaluation von Lern-erfolg als logische Rekonstruktion von Handlungen.In: Schenkel, P. (ed.) Qualitätsbeurteilung multi-medialer Lern- und Informationssysteme. Evalua-tionsmethoden auf dem Prüfstand. Nuremberg:2000, p. 262-303.

Documents

Quality in E-learning Ulf-Daniel · 2005-06-09 · Quality in E-learning The learner as a key quality as-surance category Quality will determine the future of e-learning. Many analyses