A concordance of Marshall’s e-Learning Maturity Model with Bacsich’s Pick & Mix Model

8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model

1/12

Paul Bacsich 18 February 2006

A concordance of Marshalls e-Learning Maturity Model

with Bacsichs Pick & Mix Model

Draft 1

Stephen Marshalls e-Learning Maturity Model (eMM) is being used as the methodologicalbasis of the benchmarking activity in the University of Manchester. However, since the

University of Manchesteris a member of Paul Bacsichs mini-club of four universities, all the

others of which are using the Pick & Mix methodology, it was felt useful to provide a

mapping between eMM and Pick & Mix in order to facilitate comparison and common

dialogue.

In its current form (it is due to change in the next releasesome details are known), eMM

has 43 criteria, grouped into five categories:

Learning

Development Coordination & Support

Evaluation

Organisation

Pick & Mix in its core version has 18 criteria. They are not grouped into categories but it is

likely that they could be, using for example the phases of e-learning from the Costs of

Networked Learning studies or alternatively, by using the categories of Marshall.

The next version of Pick & Mix is under constructionit will add to the core criteria and also

permit use of local criteria for an institution.

The following pages provide a concordance into the emerging new release of Pick & Mix of

each of the five categories of eMM. (See the Appendix for an indication of the current state.)

Methodology

It has to be pointed out that there are several methodological differences between eMM and

Pick & Mix. These are likely to make it hard to deploy an unmodifiedeMM in UK HE.

1. A minor one is that each eMM criterion is expressed in terms of a narrative of goodpractice whereas each Pick & Mix criterion is a more stark statement with suggestions

of how to measure it at various levels of compliance.

2. A slightly more major one is that the thrust (so far) of eMM has been at programme or

project level so that some statements have the tone of being about courses not the

whole institutionbut some rewording of these can be done if it causes confusion.

3. A much more major difference is that the scope of eMM goes into areas whichtraditionally in the UK are seen as the province of general good practice in teaching

and learning (and thus the province of QAA) rather than the province of e-learning

technology or pedagogy (and thus of JISC or HEA). Thus the scope of Pick & Mix is

considerably narrower.

4. Coupled with this is the feeling that each of the national benchmarking criteria sets

inevitably focusses on a rhetoric offeatures and issues that are seen as of more thanaverage importance. This comes out clearly below with eMM and Pick & Mix but


2/12

eMM-Pick & Mix concordance draft 1

Paul Bacsich 2 21 February 2006

similar comparisons with US methodology would show up the far greater orientation

in the US to issues of off-campus provision. And as noted in the ALT-C paper when

discussing the US Quality on the Line guidelines, Comments on these guidelines

from one group included the remarks that they were of an era and only a subset of

what is required; the other group noted that there was no real concept of governance

or legal framework in the guidelines... In several US documents there is still a rhetoricof a community of scholars, self-organising and devoid of managers and support

staff.

Note

This material is work in progress and not destined for publication in anything like its current

form. For this reason a number of niceties such as key references etc have been left out

All opinions are tentative and not yet in a fit state to be checked with parties outside those

involved in the creation of the opinions.

The next phase is to send it to the University of Manchester team for comment.

The tables

In the following tables the code n/a means that the eMM criterion is deemed not applicable

as a criterion in the Pick & Mix model; this is often because it is seen as out of scope for the

UK approach to e-learning, not that it is not important for teaching and learning.


3/12



1. Learning: Processes that directly impact on pedagogical aspects of e-learning

There are 10 processes in this category, each described below.

eMM code eMM description P&M # P&M

description

or comment

Notes

eMM.L1 Courses are designed to require students to

engage themselves in analysis, synthesis,

and evaluation as part of their course and

programme requirements

n/a This is seen in the UK as part

of good practice in teaching

and learning

eMM.L2 Student interaction with teaching staff and

other students is an essential characteristic

and is facilitated through a variety of ways



and learning

eMM.L3 Teaching staff clearly communicate how

communication channels should be used

during a course or programme



and learning

eMM.L4 Teaching staff manage studentexpectations over the type and timeliness

of responses to student communications

This is seen in the UK as partof good practice in teaching

and learning

eMM.L5 Feedback to student assignments and

questions is constructive and provided in a

timely manner

P&M93 Communic-

ation of

expectations,

and conform-

ance to them,

of the assign-

ment hand-

ling and

feedback

system

This is seen in the UK as part


and learning but there are

specific aspects for e-

learning, thus treated as a

P&M supplementary criterion

eMM.L6 Students are instructed in the proper

methods of effective research, including

assessment of the validity of resources

n/a Part of the general

information literacy issue

needed for all students

whether on e-learning

courses or not

eMM.L7

(now goes

with L1)

Learning outcomes for each course are

summarised in a clearly written,

straightforward statement



and learning

eMM.L8 Assessment of students communicates high

expectations



and learningeMM.L9 Student work is subject to clearly

communicated timetables and deadlines

P&M93? See the comments on L5; but

note that much experience in

distance learning shows that

rigid deadlines increase drop-

out

eMM.L10 Courses are designed to support a diversity

of learning styles and to ensure

accessibility

P&M05 Accessibility Note that in the UK it is usual

to treat this separately for

legal reasons; thus L10 has a

wider scope


4/12



2. Development: Processes surrounding the creation and maintenance of e-learning

resources



description

or comment

Notes

eMM.D1 Guidelines regarding minimum standards

are used for course development, design

and delivery

P&M09 Learning

material

P&M09 is only part of the

scope of eMM.D1

eMM.D2 The reliability of the technology delivery

system is as failsafe as possible

P&M52 Underpinning

IT/comms

reliability

This is a supplementary P&M

criterion

eMM.D3 Learning outcomes, not the availability of

existing technology, determine the

technology being used to deliver course

content

n/a - Many in the UK would

regard this as unrealistic,

given university financing

and student demographicseMM.D4 Technical assistance in course development

is available to teaching staff

P&M16 Technical

assistance to

academic

staff

Perhaps both could be

criticised as focussing purely

on academic staff

eMM.D5 Teaching staff are encouraged to use

technical assistance when (re)developing

courses

P&M16 Technical

assistance to

academic

staff

This is part of P&M16

eMM.D6 Teaching staff members are assisted in the

transition from classroom teaching to

online instruction

P&M07? Instructional

design and

pedagogy

Perhaps one should add a

criterion on pedagogic

support, or (more

controversially) extend

technical support to

techno-pedagogic support


5/12



3. Coordination & Support: Processes around the day-to-day management and

support of e-learning delivery



description

or comment

Notes

eMM.C1 A centralised system provides support for

building and maintaining the e-learning

infrastructure

(Marshall recommends moving to Dthis

seems hard to justify)

P&M15 Organisation

(to support e-

learning)

In the UK it is by and large

accepted that the technical

infrastructure should be

centralised, but not for the

pedagogic infrastructure

eMM.C2 Students have access to sufficient library

resources that may include a virtual

library accessible through the World Wide

Web



and learning

eMM.C3 Teaching staff and students agree uponexpectations regarding times for student

assignment completion and staff response

P&M93 This seems too closely linkedto L5 and L9 to be treated

separately

eMM.C4 Students are provided with hands-on

training and information to aid them in

securing material from a range of sources

consistent with the discipline or subject



and learning

eMM.C5 Students have convenient access to

technical assistance throughout the duration

of the course/programme

P&M92 Competence

and

timeliness of

help from

Help Desk

For some reason this did not

make it into the P&M core

criteria

eMM.C6 Students are provided with detailed

instructions regarding the electronic media

used in a course prior to commencing it

n/a Normally done in the last 10

years; so not a differentiator

eMM.C7 Students are able to practice with any

technologies prior to commencing a course

n/a This is an important point but

usually in the UK the practice

time is very limited due to the

rush at the beginning of

semesters

eMM.C8 Questions directed to student service

personnel are answered accurately and

quickly

P&M92 Links to C5; indeed Marshall

recommends conflating them

eMM.C9 A structured system is in place to addressstudent complaints

n/a This is seen in the UK as partof good practice in running a

university

eMM.C10 Instructor training and assistance continues

throughout the online course

P&M10 Training

eMM.C11 Teaching staff are provided with support

resources to deal with issues arising from

student use of electronically-accessed data

P&M65 Level, skill

and delicacy

of

management

of plagiarism

issues

This may deconstruct into a

variety of criteria


6/12



4. Evaluation: Processes surrounding the evaluation and quality control of e-

learning throughout its entire lifecycle



description

or comment

Notes

eMM.E1 The programmes educational effectiveness

is formatively and summatively assessed

with multiple, standards based, and

independent evaluations

P&M14 Evaluation Marshall has recommended

conflating these four criteria

eMM.E2 The programmes teaching/learning

process is formatively and summatively

assessed with multiple, standards based,

and independent evaluations

eMM.E3 Summative data such as enrolment

numbers, completion rates, and costing isused as a measure of effectiveness within

course/programmes

eMM.E4 Success of technology/innovation used as a

measure of effectiveness within

course/programmes

eMM.E5 Intended learning outcomes are reviewed

regularly to ensure clarity, utility, and

appropriateness

n/a This is seen

in the UK as

part of good

practice in

teaching and

learning

eMM.E6. Instructional materials are reviewed

periodically to ensure they meet

programme standards

n/a This is seen

in the UK as

part of good

practice in

teaching and

learning

eMM.E7 Teaching staff capability in making the

transition from classroom to online

teaching is formally assessed during

training

n/a This is related

to P&M10

but is not

believed in

the UK to be

either feasible

or correlatedwith success

in e-learning


7/12



5. Organisation: Processes surrounding the creation and maintenance of e-learning

resources



description

or comment

Notes

eMM.O1 A documented set of formal criteria are

used to determine access to funding and

other resources which support course and

programme (re)development

P&M07 Decision-

making

It is not the set of formal

criteria so much as the

processes surrounding it,

including whether such a set

is taken seriously

eMM.O2 A documented technology plan is in place

and operational to ensure quality of

delivery standards

P&M06 e-learning

strategy

In the UK the normal practice

is to regard a technology plan

as only part of an e-learning

strategy and subservient to a

pedagogic strategyin eMM

the pedagogic strategy seems

implicit

eMM.O3 A documented technology plan is in place

and operational to ensure the integrity and

validity of information delivered, collected

and stored

n/a Important but not normally in

the UK seen as central to e-

learning success

eMM.O4 Before starting a programme, students are

advised of any particular requirements of

that programme to ensure they possess the

personal and technical skills needed for

that programme

P&M91 Communicat-

ion to

students of

expectations

of students

eMM.O5 Students are provided with supplementalcourse information that outlines course

objectives, concepts and ideas

n/a This is seen in the UK as partof good practice in teaching

and learning

eMM.O6 Students are provided with supplemental

course information that outlines admission

requirements, tuition and fees and other

relevant administration information

n/a (Marshall recommends

merging these two.)

Both are seen in the UK as

part of good practice in

teaching and learningin

fact, as no-brainerseMM.O7 Students are provided with supplemental

course information that outlines

requirements for additional resources such

as books or other materials

eMM.O8 Students are provided with supplementalcourse information that outlines student

support services.

n/a This is seen in the UK as partof good practice in teaching

and learning and university

administration

eMM.O9 Before starting a programme, students are

advised of any particular technological

requirements of that programme to ensure

they have access to the minimal technology

required by the course design

P&M91? Good idea; see also P&M95


8/12



6. Conclusions

From this first cut it does not look as though the match between eMM and Pick & Mix is

particularly close.

However, as Marshall notes in his section on Future Development:

He has cut down his set of 43 processes to 34 by skilful compositing (we have takenthis into account where it is was easy to do so without distorting his original

sequencing)

A review of the literature suggests that as many as one hundred additional processes

could potentially be incorporated.

It should also be noted that in terms of Pick & Mix that:

the current core Pick & Mix of 18 criteria was in fact shrunk from a set of around 25

many other criteria are under consideration (see the Appendix), perhaps up to 20 more

Thus we may not yet be matching those methodologies in a mature enough state.

In terms of both methodologies, it should be noted that they both claim to have common

grounding in archetypal best practice compendia such as Quality on the Line, The Seven

Principles, etc. Perhaps along the lines of their developments they have interpreted these in

different ways. Some earlier work which was stopped when the current phase of funded

benchmarking activity started may have to be reviewed and refinedthis provided both exit

strategies for the various archetypal criteria and (in reverse) a pedigree for Pick & Mix. The

early literature search review (the Middlesex paper

http://www.cs.mdx.ac.uk/staff/profiles/p_bacsich/Benchmark-theory.pdf) contains some of

this but for the usual reasons later published papers were more concise.A natural way forward is to rationally reconstruct a UK-eMM basing it initially on Pick &

Mix but taking care to check out other UK benchmarking, best practice and developmental

traditions (such as NLN and ELTI)noting, however, that Pick & Mix already incorporates

many of these ideas. Such a UK-eMM is an admirable research topic and would also be

practically very useful as a kind of ground penetrating radar compared with the alleged

shallower scan of Pick & Mix. but it would lose the benefit of international comparability

however, unlike with costings, that international comparability may remain a dream. And at

this stage eMM is deployed in full only in New ZealandAustralia is still taking it slowly in

benchmarking.\
http://www.cs.mdx.ac.uk/staff/profiles/p_bacsich/Benchmark-theory.pdfhttp://www.cs.mdx.ac.uk/staff/profiles/p_bacsich/Benchmark-theory.pdfhttp://www.cs.mdx.ac.uk/staff/profiles/p_bacsich/Benchmark-theory.pdf


9/12



Appendix: The emerging Pick & Mix extended criteria

(This is adapted from a working paper done at the instigation of the University of Leicester.

Further extensions are being done as other institutions engage.)

Core criteria

The following version of the standard Pick & Mix table is simplified, leaving out levels 2, 4,

and 6 (excellence), in order to fit more neatly and readably onto an A4 portrait page.

P&M Factor 1 3 5 Instrument

01 Adoption phase

overall (Rogers)

Innovators only Early majority

taking it up

All taken it up

except some

laggards

Interviews,

surveys,

documentation in

IT reports, etc.

02 VLE stage No VLE VLEs reducing in

number to around

two

One VLE Observation,

purchase orders

03 Tools use No use of tools

beyond email, Web

and the VLE

minimum set

Widespread use of

at least one specific

tool, e.g.

assignment

handling, CAA

HEI-wide use of

several tools

Interviews, cross-

checking with JISC

and CETIS, etc.

04 IT underpinning

usability

No usability

testing, no grasp of

the concept

Explicit usability

testing of all key

systems

All services usable,

with internal

evidence to back

this up

(Further advice is

needed from

UKERNA, JISC

and UCISA)

05 Accessibility e-learning material

and services is not

accessible

Almost all e-

learning material

and services

conform to

minimum standardsof accessibility

e-learning material

and services are

accessible, and key

components

validated byexternal agencies

(Split off separately

for legal reasons)

06 e-Learning

Strategy

No e-Learning

Strategy. No recent

Learning and

Teaching Strategy

e-Learning

Strategy produced

from time to time,

e.g. under pressure

from HEFCE or for

particular grants

Regularly updated

e-Learning

Strategy, integrated

with Learning and

Teaching Strategy

and all related

strategies

Review of HEFCE,

TQEF and other

documents;

interview with

PVC responsible

07 Decision-making No decision

making regarding

e-learningeach

project is different

E-learning

decisions (e.g. for

VLEs) get taken

but take a long

time and arecontested even

after the decision is

taken

Effective decision-

making for e-

learning across the

whole institution,

includingvariations when

justified

Observation and

perusal of papers

08 Instructional

Design/ Pedagogy

Terms not

understood in the

HEI

Terms well

understood within

the learning and

teaching centre and

among some

academic staff

Pedagogic

guidelines for the

whole HEI, and

acted on

Interviews


10/12




09 Learning material Little conformance

of learning material

to house style for

editing or layout

Most learning

material conforms

to explicit editorial

and layout

guidelines

HEI-wide

standards for

learning material,

which are adhered

to and embedded at

any early stage, e.g.by style sheets

Perusal of material,

interviews

10 Training No systematic

training for e-

learning

HEI-wide training

programme set up

but little

monitoring of

attendance or

encouragement to

go

All staff trained in

VLE use,

appropriate to job

typeand

retrained when

needed

%ages plus

narrative (this may

not involve training

courses; but is

likely to)

11 Academic

workload

No allowance

given for the

different workload

pattern of e-

learning courses

A work planning

system which

makes some

attempt to cope,

however crudely,with e-learning

courses

Work planning

system which

recognises the

main differences

that e-learningcourses have from

traditional

Detailed and

possibly

anonymous

interviews and

questionnairessome union

sensitivities likely

in some HEIs

12 Costs No understanding

of costs of e-

learning except

possibly in (maybe)

the business school

Good informal

understanding of

costs in faculties

and central

departments but

not coordinated

Activity-Based

Costing being used

in part

Interviews and

questionnaires

(the basis here is

from CNL and

INSIGHT JISC

projects, also Becta

TCO)

13 Planning No planning

process for e-

learning offerings

Systematic

planning process

for e-learning but

unintegrated with

related planning

such as for IT or

for space

Integrated planning

process for e-

learning within

overall course

planning and some

links to IT and

space planning

Interviews and

questionnaires

(more work to be

done on levels

process models

may help)

14 Evaluation No evaluation of

courses take place

that is done by

evaluation

professionals

Evaluation of key

courses is done

from time to time,

by professionals

Regular evaluation

of all courses using

a variety of

measurement

techniques and

involving outside

agencies where

appropriate

Interviews with key

evaluators; perusal

of conference and

journal papers

15 Organisation No appointments

of e-learning staff

Central unit or sub-

unit set up to

support e-learning

developments

Central unit has

Director-level

university manager

in charge and links

to support teams in

faculties

Interview with VC

and relevant

PVC(s)

16 Technical support

to academic staff

No specific

technical support

for the typical

(unfunded)

academic engaged

in e-learning

Key staff engaged

in the main e-

learning projects

are well supported

by technical staff

All staff engaged in

e-learning process

have nearby fast-

response tech

support

Interview with both

top-level staff and

selective interviews

with grass-roots

staff


11/12




17 Quality and

Excellence

Conformance to

QAA in a

minimalist way

Conformance to

QAA precepts

including those

that impinge on e-

learning

Adoption of some

appropriate quality

methodology

(EFQM, etc)

integrated with

course qualitymechanisms

derived from QAA

precepts

Interviews,

questionnaires,

quality reviews,

etc.

18 Staff recognition

for e-learning

No recognition for

staff, explicit

pressure against

(e.g. due to RAE)

Some recognition

such as Teaching

Fellows within an

essentially RAE-

driven paradigm

Staff engaged only

in or supporting the

teaching process

can reach high

levels of salary and

responsibility

Documentary

evidence

Extension criteria

Extension criteria have at present been assigned to a tentative numbering system, which will

change once the choice of criteria has stabilised.

Seven cr iter ia that were in the beta version of Pick & M ix that some want to bring back

P&M Factor Notes

51 Overall NLN-style ILT adoption level Only of interest for HE-FE consortia

52 Overall eMM adoption level Only of interest for international consortia

52 Underpinning IT/comms reliability Too IT-oriented? But it is a critical success factor so

likely to stay.

54 Underpinning IT/comms performance Too IT-oriented? But it is a critical success factor so

likely to stay (and maybe in new core)

55 Foresight into technology and pedagogy Specialised but likely to stay (though not core)

56 Collaboration But is more collaboration better? So how to score?

57 Competence in managing IPR for e-learning Specialised but likely to stay (though not core)

Additi onal instituti on-ori ented extension cr iter ia under consideration at more than one

HEI

This list has benefited from the authors notes of the presentations at the Launch Meeting

but a second pass through the presentations is required to refine this.

58 Market research Not a strength of some UK universities. Likely to stay

(and in the new core)59 Competitor research Often ignored. Likely to stay (but not core).

60 Competence and frequency of benchmarking Too internal a criterion?

61 Public relations external and communications

internally with stakeholders

62 Level of integration of e-learning with admin

systems

The whole MLE versus VLE debate

63 Level of exploitation of pre-existing student

IT skills

The NetGen issue. Fashionable and likely to stay;

64 Level of development of Computer Aided

Assessment

Specialised; likely to stay but not in new core.


12/12



65 Level, skill and delicacy of management of

plagiarism

Specialised; likely to stay but not in new core

66 Contribution of e-learning to WP agenda Hard to tease out? Important to some HEIs.

66 Level of integration/interaction of physical

space management with management of IT

67` Effective management of risks associated

with e-learning

A component of the overall HEI Risk Register

68 Amount and level of research leveraged on e-

learning

Important to some HEIs but not many; likely to stay;

but not in new core.

69 Cost-effectiveness of e-learning A tough one. Likely to staycoming back into fashion

in Australia and never out of fashion in US

Additi onal student-or iented cri ter ia

91 Communication of expectations of students

to students

92 Competence and timeliness of IT help from

Help Desk

93 Communication of expectations and

conformance to them of the assignment

handling and feedback system

94 Overall student satisfaction with e-learning A component of overall student satisfaction

95 Understanding and management of the

student-borne costs of e-learning

99 Convergence of functionality and usability of

IT-based systems of technical, academic and

pastoral help with face to face systems

Needs much discussion.

Documents

A concordance of Marshall’s e-Learning Maturity Model with Bacsich’s Pick & Mix Model