8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
1/12
Paul Bacsich 18 February 2006
A concordance of Marshalls e-Learning Maturity Model
with Bacsichs Pick & Mix Model
Draft 1
Stephen Marshalls e-Learning Maturity Model (eMM) is being used as the methodologicalbasis of the benchmarking activity in the University of Manchester. However, since the
University of Manchesteris a member of Paul Bacsichs mini-club of four universities, all the
others of which are using the Pick & Mix methodology, it was felt useful to provide a
mapping between eMM and Pick & Mix in order to facilitate comparison and common
dialogue.
In its current form (it is due to change in the next releasesome details are known), eMM
has 43 criteria, grouped into five categories:
Learning
Development Coordination & Support
Evaluation
Organisation
Pick & Mix in its core version has 18 criteria. They are not grouped into categories but it is
likely that they could be, using for example the phases of e-learning from the Costs of
Networked Learning studies or alternatively, by using the categories of Marshall.
The next version of Pick & Mix is under constructionit will add to the core criteria and also
permit use of local criteria for an institution.
The following pages provide a concordance into the emerging new release of Pick & Mix of
each of the five categories of eMM. (See the Appendix for an indication of the current state.)
Methodology
It has to be pointed out that there are several methodological differences between eMM and
Pick & Mix. These are likely to make it hard to deploy an unmodifiedeMM in UK HE.
1. A minor one is that each eMM criterion is expressed in terms of a narrative of goodpractice whereas each Pick & Mix criterion is a more stark statement with suggestions
of how to measure it at various levels of compliance.
2. A slightly more major one is that the thrust (so far) of eMM has been at programme or
project level so that some statements have the tone of being about courses not the
whole institutionbut some rewording of these can be done if it causes confusion.
3. A much more major difference is that the scope of eMM goes into areas whichtraditionally in the UK are seen as the province of general good practice in teaching
and learning (and thus the province of QAA) rather than the province of e-learning
technology or pedagogy (and thus of JISC or HEA). Thus the scope of Pick & Mix is
considerably narrower.
4. Coupled with this is the feeling that each of the national benchmarking criteria sets
inevitably focusses on a rhetoric offeatures and issues that are seen as of more thanaverage importance. This comes out clearly below with eMM and Pick & Mix but
8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
2/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 2 21 February 2006
similar comparisons with US methodology would show up the far greater orientation
in the US to issues of off-campus provision. And as noted in the ALT-C paper when
discussing the US Quality on the Line guidelines, Comments on these guidelines
from one group included the remarks that they were of an era and only a subset of
what is required; the other group noted that there was no real concept of governance
or legal framework in the guidelines... In several US documents there is still a rhetoricof a community of scholars, self-organising and devoid of managers and support
staff.
Note
This material is work in progress and not destined for publication in anything like its current
form. For this reason a number of niceties such as key references etc have been left out
All opinions are tentative and not yet in a fit state to be checked with parties outside those
involved in the creation of the opinions.
The next phase is to send it to the University of Manchester team for comment.
The tables
In the following tables the code n/a means that the eMM criterion is deemed not applicable
as a criterion in the Pick & Mix model; this is often because it is seen as out of scope for the
UK approach to e-learning, not that it is not important for teaching and learning.
8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
3/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 3 21 February 2006
1. Learning: Processes that directly impact on pedagogical aspects of e-learning
There are 10 processes in this category, each described below.
eMM code eMM description P&M # P&M
description
or comment
Notes
eMM.L1 Courses are designed to require students to
engage themselves in analysis, synthesis,
and evaluation as part of their course and
programme requirements
n/a This is seen in the UK as part
of good practice in teaching
and learning
eMM.L2 Student interaction with teaching staff and
other students is an essential characteristic
and is facilitated through a variety of ways
n/a This is seen in the UK as part
of good practice in teaching
and learning
eMM.L3 Teaching staff clearly communicate how
communication channels should be used
during a course or programme
n/a This is seen in the UK as part
of good practice in teaching
and learning
eMM.L4 Teaching staff manage studentexpectations over the type and timeliness
of responses to student communications
This is seen in the UK as partof good practice in teaching
and learning
eMM.L5 Feedback to student assignments and
questions is constructive and provided in a
timely manner
P&M93 Communic-
ation of
expectations,
and conform-
ance to them,
of the assign-
ment hand-
ling and
feedback
system
This is seen in the UK as part
of good practice in teaching
and learning but there are
specific aspects for e-
learning, thus treated as a
P&M supplementary criterion
eMM.L6 Students are instructed in the proper
methods of effective research, including
assessment of the validity of resources
n/a Part of the general
information literacy issue
needed for all students
whether on e-learning
courses or not
eMM.L7
(now goes
with L1)
Learning outcomes for each course are
summarised in a clearly written,
straightforward statement
n/a This is seen in the UK as part
of good practice in teaching
and learning
eMM.L8 Assessment of students communicates high
expectations
n/a This is seen in the UK as part
of good practice in teaching
and learningeMM.L9 Student work is subject to clearly
communicated timetables and deadlines
P&M93? See the comments on L5; but
note that much experience in
distance learning shows that
rigid deadlines increase drop-
out
eMM.L10 Courses are designed to support a diversity
of learning styles and to ensure
accessibility
P&M05 Accessibility Note that in the UK it is usual
to treat this separately for
legal reasons; thus L10 has a
wider scope
8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
4/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 4 21 February 2006
2. Development: Processes surrounding the creation and maintenance of e-learning
resources
There are 6 processes in this category, each described below.
eMM code eMM description P&M # P&M
description
or comment
Notes
eMM.D1 Guidelines regarding minimum standards
are used for course development, design
and delivery
P&M09 Learning
material
P&M09 is only part of the
scope of eMM.D1
eMM.D2 The reliability of the technology delivery
system is as failsafe as possible
P&M52 Underpinning
IT/comms
reliability
This is a supplementary P&M
criterion
eMM.D3 Learning outcomes, not the availability of
existing technology, determine the
technology being used to deliver course
content
n/a - Many in the UK would
regard this as unrealistic,
given university financing
and student demographicseMM.D4 Technical assistance in course development
is available to teaching staff
P&M16 Technical
assistance to
academic
staff
Perhaps both could be
criticised as focussing purely
on academic staff
eMM.D5 Teaching staff are encouraged to use
technical assistance when (re)developing
courses
P&M16 Technical
assistance to
academic
staff
This is part of P&M16
eMM.D6 Teaching staff members are assisted in the
transition from classroom teaching to
online instruction
P&M07? Instructional
design and
pedagogy
Perhaps one should add a
criterion on pedagogic
support, or (more
controversially) extend
technical support to
techno-pedagogic support
8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
5/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 5 21 February 2006
3. Coordination & Support: Processes around the day-to-day management and
support of e-learning delivery
There are 11 processes in this category, each described below.
eMM code eMM description P&M # P&M
description
or comment
Notes
eMM.C1 A centralised system provides support for
building and maintaining the e-learning
infrastructure
(Marshall recommends moving to Dthis
seems hard to justify)
P&M15 Organisation
(to support e-
learning)
In the UK it is by and large
accepted that the technical
infrastructure should be
centralised, but not for the
pedagogic infrastructure
eMM.C2 Students have access to sufficient library
resources that may include a virtual
library accessible through the World Wide
Web
n/a This is seen in the UK as part
of good practice in teaching
and learning
eMM.C3 Teaching staff and students agree uponexpectations regarding times for student
assignment completion and staff response
P&M93 This seems too closely linkedto L5 and L9 to be treated
separately
eMM.C4 Students are provided with hands-on
training and information to aid them in
securing material from a range of sources
consistent with the discipline or subject
n/a This is seen in the UK as part
of good practice in teaching
and learning
eMM.C5 Students have convenient access to
technical assistance throughout the duration
of the course/programme
P&M92 Competence
and
timeliness of
help from
Help Desk
For some reason this did not
make it into the P&M core
criteria
eMM.C6 Students are provided with detailed
instructions regarding the electronic media
used in a course prior to commencing it
n/a Normally done in the last 10
years; so not a differentiator
eMM.C7 Students are able to practice with any
technologies prior to commencing a course
n/a This is an important point but
usually in the UK the practice
time is very limited due to the
rush at the beginning of
semesters
eMM.C8 Questions directed to student service
personnel are answered accurately and
quickly
P&M92 Links to C5; indeed Marshall
recommends conflating them
eMM.C9 A structured system is in place to addressstudent complaints
n/a This is seen in the UK as partof good practice in running a
university
eMM.C10 Instructor training and assistance continues
throughout the online course
P&M10 Training
eMM.C11 Teaching staff are provided with support
resources to deal with issues arising from
student use of electronically-accessed data
P&M65 Level, skill
and delicacy
of
management
of plagiarism
issues
This may deconstruct into a
variety of criteria
8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
6/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 6 21 February 2006
4. Evaluation: Processes surrounding the evaluation and quality control of e-
learning throughout its entire lifecycle
There are 7 processes in this category, each described below.
eMM code eMM description P&M # P&M
description
or comment
Notes
eMM.E1 The programmes educational effectiveness
is formatively and summatively assessed
with multiple, standards based, and
independent evaluations
P&M14 Evaluation Marshall has recommended
conflating these four criteria
eMM.E2 The programmes teaching/learning
process is formatively and summatively
assessed with multiple, standards based,
and independent evaluations
eMM.E3 Summative data such as enrolment
numbers, completion rates, and costing isused as a measure of effectiveness within
course/programmes
eMM.E4 Success of technology/innovation used as a
measure of effectiveness within
course/programmes
eMM.E5 Intended learning outcomes are reviewed
regularly to ensure clarity, utility, and
appropriateness
n/a This is seen
in the UK as
part of good
practice in
teaching and
learning
eMM.E6. Instructional materials are reviewed
periodically to ensure they meet
programme standards
n/a This is seen
in the UK as
part of good
practice in
teaching and
learning
eMM.E7 Teaching staff capability in making the
transition from classroom to online
teaching is formally assessed during
training
n/a This is related
to P&M10
but is not
believed in
the UK to be
either feasible
or correlatedwith success
in e-learning
8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
7/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 7 21 February 2006
5. Organisation: Processes surrounding the creation and maintenance of e-learning
resources
There are 9 processes in this category, each described below.
eMM code eMM description P&M # P&M
description
or comment
Notes
eMM.O1 A documented set of formal criteria are
used to determine access to funding and
other resources which support course and
programme (re)development
P&M07 Decision-
making
It is not the set of formal
criteria so much as the
processes surrounding it,
including whether such a set
is taken seriously
eMM.O2 A documented technology plan is in place
and operational to ensure quality of
delivery standards
P&M06 e-learning
strategy
In the UK the normal practice
is to regard a technology plan
as only part of an e-learning
strategy and subservient to a
pedagogic strategyin eMM
the pedagogic strategy seems
implicit
eMM.O3 A documented technology plan is in place
and operational to ensure the integrity and
validity of information delivered, collected
and stored
n/a Important but not normally in
the UK seen as central to e-
learning success
eMM.O4 Before starting a programme, students are
advised of any particular requirements of
that programme to ensure they possess the
personal and technical skills needed for
that programme
P&M91 Communicat-
ion to
students of
expectations
of students
eMM.O5 Students are provided with supplementalcourse information that outlines course
objectives, concepts and ideas
n/a This is seen in the UK as partof good practice in teaching
and learning
eMM.O6 Students are provided with supplemental
course information that outlines admission
requirements, tuition and fees and other
relevant administration information
n/a (Marshall recommends
merging these two.)
Both are seen in the UK as
part of good practice in
teaching and learningin
fact, as no-brainerseMM.O7 Students are provided with supplemental
course information that outlines
requirements for additional resources such
as books or other materials
eMM.O8 Students are provided with supplementalcourse information that outlines student
support services.
n/a This is seen in the UK as partof good practice in teaching
and learning and university
administration
eMM.O9 Before starting a programme, students are
advised of any particular technological
requirements of that programme to ensure
they have access to the minimal technology
required by the course design
P&M91? Good idea; see also P&M95
8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
8/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 8 21 February 2006
6. Conclusions
From this first cut it does not look as though the match between eMM and Pick & Mix is
particularly close.
However, as Marshall notes in his section on Future Development:
He has cut down his set of 43 processes to 34 by skilful compositing (we have takenthis into account where it is was easy to do so without distorting his original
sequencing)
A review of the literature suggests that as many as one hundred additional processes
could potentially be incorporated.
It should also be noted that in terms of Pick & Mix that:
the current core Pick & Mix of 18 criteria was in fact shrunk from a set of around 25
many other criteria are under consideration (see the Appendix), perhaps up to 20 more
Thus we may not yet be matching those methodologies in a mature enough state.
In terms of both methodologies, it should be noted that they both claim to have common
grounding in archetypal best practice compendia such as Quality on the Line, The Seven
Principles, etc. Perhaps along the lines of their developments they have interpreted these in
different ways. Some earlier work which was stopped when the current phase of funded
benchmarking activity started may have to be reviewed and refinedthis provided both exit
strategies for the various archetypal criteria and (in reverse) a pedigree for Pick & Mix. The
early literature search review (the Middlesex paper
http://www.cs.mdx.ac.uk/staff/profiles/p_bacsich/Benchmark-theory.pdf) contains some of
this but for the usual reasons later published papers were more concise.A natural way forward is to rationally reconstruct a UK-eMM basing it initially on Pick &
Mix but taking care to check out other UK benchmarking, best practice and developmental
traditions (such as NLN and ELTI)noting, however, that Pick & Mix already incorporates
many of these ideas. Such a UK-eMM is an admirable research topic and would also be
practically very useful as a kind of ground penetrating radar compared with the alleged
shallower scan of Pick & Mix. but it would lose the benefit of international comparability
however, unlike with costings, that international comparability may remain a dream. And at
this stage eMM is deployed in full only in New ZealandAustralia is still taking it slowly in
benchmarking.\
http://www.cs.mdx.ac.uk/staff/profiles/p_bacsich/Benchmark-theory.pdfhttp://www.cs.mdx.ac.uk/staff/profiles/p_bacsich/Benchmark-theory.pdfhttp://www.cs.mdx.ac.uk/staff/profiles/p_bacsich/Benchmark-theory.pdf8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
9/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 9 21 February 2006
Appendix: The emerging Pick & Mix extended criteria
(This is adapted from a working paper done at the instigation of the University of Leicester.
Further extensions are being done as other institutions engage.)
Core criteria
The following version of the standard Pick & Mix table is simplified, leaving out levels 2, 4,
and 6 (excellence), in order to fit more neatly and readably onto an A4 portrait page.
P&M Factor 1 3 5 Instrument
01 Adoption phase
overall (Rogers)
Innovators only Early majority
taking it up
All taken it up
except some
laggards
Interviews,
surveys,
documentation in
IT reports, etc.
02 VLE stage No VLE VLEs reducing in
number to around
two
One VLE Observation,
purchase orders
03 Tools use No use of tools
beyond email, Web
and the VLE
minimum set
Widespread use of
at least one specific
tool, e.g.
assignment
handling, CAA
HEI-wide use of
several tools
Interviews, cross-
checking with JISC
and CETIS, etc.
04 IT underpinning
usability
No usability
testing, no grasp of
the concept
Explicit usability
testing of all key
systems
All services usable,
with internal
evidence to back
this up
(Further advice is
needed from
UKERNA, JISC
and UCISA)
05 Accessibility e-learning material
and services is not
accessible
Almost all e-
learning material
and services
conform to
minimum standardsof accessibility
e-learning material
and services are
accessible, and key
components
validated byexternal agencies
(Split off separately
for legal reasons)
06 e-Learning
Strategy
No e-Learning
Strategy. No recent
Learning and
Teaching Strategy
e-Learning
Strategy produced
from time to time,
e.g. under pressure
from HEFCE or for
particular grants
Regularly updated
e-Learning
Strategy, integrated
with Learning and
Teaching Strategy
and all related
strategies
Review of HEFCE,
TQEF and other
documents;
interview with
PVC responsible
07 Decision-making No decision
making regarding
e-learningeach
project is different
E-learning
decisions (e.g. for
VLEs) get taken
but take a long
time and arecontested even
after the decision is
taken
Effective decision-
making for e-
learning across the
whole institution,
includingvariations when
justified
Observation and
perusal of papers
08 Instructional
Design/ Pedagogy
Terms not
understood in the
HEI
Terms well
understood within
the learning and
teaching centre and
among some
academic staff
Pedagogic
guidelines for the
whole HEI, and
acted on
Interviews
8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
10/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 10 21 February 2006
P&M Factor 1 3 5 Instrument
09 Learning material Little conformance
of learning material
to house style for
editing or layout
Most learning
material conforms
to explicit editorial
and layout
guidelines
HEI-wide
standards for
learning material,
which are adhered
to and embedded at
any early stage, e.g.by style sheets
Perusal of material,
interviews
10 Training No systematic
training for e-
learning
HEI-wide training
programme set up
but little
monitoring of
attendance or
encouragement to
go
All staff trained in
VLE use,
appropriate to job
typeand
retrained when
needed
%ages plus
narrative (this may
not involve training
courses; but is
likely to)
11 Academic
workload
No allowance
given for the
different workload
pattern of e-
learning courses
A work planning
system which
makes some
attempt to cope,
however crudely,with e-learning
courses
Work planning
system which
recognises the
main differences
that e-learningcourses have from
traditional
Detailed and
possibly
anonymous
interviews and
questionnairessome union
sensitivities likely
in some HEIs
12 Costs No understanding
of costs of e-
learning except
possibly in (maybe)
the business school
Good informal
understanding of
costs in faculties
and central
departments but
not coordinated
Activity-Based
Costing being used
in part
Interviews and
questionnaires
(the basis here is
from CNL and
INSIGHT JISC
projects, also Becta
TCO)
13 Planning No planning
process for e-
learning offerings
Systematic
planning process
for e-learning but
unintegrated with
related planning
such as for IT or
for space
Integrated planning
process for e-
learning within
overall course
planning and some
links to IT and
space planning
Interviews and
questionnaires
(more work to be
done on levels
process models
may help)
14 Evaluation No evaluation of
courses take place
that is done by
evaluation
professionals
Evaluation of key
courses is done
from time to time,
by professionals
Regular evaluation
of all courses using
a variety of
measurement
techniques and
involving outside
agencies where
appropriate
Interviews with key
evaluators; perusal
of conference and
journal papers
15 Organisation No appointments
of e-learning staff
Central unit or sub-
unit set up to
support e-learning
developments
Central unit has
Director-level
university manager
in charge and links
to support teams in
faculties
Interview with VC
and relevant
PVC(s)
16 Technical support
to academic staff
No specific
technical support
for the typical
(unfunded)
academic engaged
in e-learning
Key staff engaged
in the main e-
learning projects
are well supported
by technical staff
All staff engaged in
e-learning process
have nearby fast-
response tech
support
Interview with both
top-level staff and
selective interviews
with grass-roots
staff
8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
11/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 11 21 February 2006
P&M Factor 1 3 5 Instrument
17 Quality and
Excellence
Conformance to
QAA in a
minimalist way
Conformance to
QAA precepts
including those
that impinge on e-
learning
Adoption of some
appropriate quality
methodology
(EFQM, etc)
integrated with
course qualitymechanisms
derived from QAA
precepts
Interviews,
questionnaires,
quality reviews,
etc.
18 Staff recognition
for e-learning
No recognition for
staff, explicit
pressure against
(e.g. due to RAE)
Some recognition
such as Teaching
Fellows within an
essentially RAE-
driven paradigm
Staff engaged only
in or supporting the
teaching process
can reach high
levels of salary and
responsibility
Documentary
evidence
Extension criteria
Extension criteria have at present been assigned to a tentative numbering system, which will
change once the choice of criteria has stabilised.
Seven cr iter ia that were in the beta version of Pick & M ix that some want to bring back
P&M Factor Notes
51 Overall NLN-style ILT adoption level Only of interest for HE-FE consortia
52 Overall eMM adoption level Only of interest for international consortia
52 Underpinning IT/comms reliability Too IT-oriented? But it is a critical success factor so
likely to stay.
54 Underpinning IT/comms performance Too IT-oriented? But it is a critical success factor so
likely to stay (and maybe in new core)
55 Foresight into technology and pedagogy Specialised but likely to stay (though not core)
56 Collaboration But is more collaboration better? So how to score?
57 Competence in managing IPR for e-learning Specialised but likely to stay (though not core)
Additi onal instituti on-ori ented extension cr iter ia under consideration at more than one
HEI
This list has benefited from the authors notes of the presentations at the Launch Meeting
but a second pass through the presentations is required to refine this.
58 Market research Not a strength of some UK universities. Likely to stay
(and in the new core)59 Competitor research Often ignored. Likely to stay (but not core).
60 Competence and frequency of benchmarking Too internal a criterion?
61 Public relations external and communications
internally with stakeholders
62 Level of integration of e-learning with admin
systems
The whole MLE versus VLE debate
63 Level of exploitation of pre-existing student
IT skills
The NetGen issue. Fashionable and likely to stay;
64 Level of development of Computer Aided
Assessment
Specialised; likely to stay but not in new core.
8/22/2019 A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model
12/12
eMM-Pick & Mix concordance draft 1
Paul Bacsich 12 21 February 2006
65 Level, skill and delicacy of management of
plagiarism
Specialised; likely to stay but not in new core
66 Contribution of e-learning to WP agenda Hard to tease out? Important to some HEIs.
66 Level of integration/interaction of physical
space management with management of IT
67` Effective management of risks associated
with e-learning
A component of the overall HEI Risk Register
68 Amount and level of research leveraged on e-
learning
Important to some HEIs but not many; likely to stay;
but not in new core.
69 Cost-effectiveness of e-learning A tough one. Likely to staycoming back into fashion
in Australia and never out of fashion in US
Additi onal student-or iented cri ter ia
91 Communication of expectations of students
to students
92 Competence and timeliness of IT help from
Help Desk
93 Communication of expectations and
conformance to them of the assignment
handling and feedback system
94 Overall student satisfaction with e-learning A component of overall student satisfaction
95 Understanding and management of the
student-borne costs of e-learning
99 Convergence of functionality and usability of
IT-based systems of technical, academic and
pastoral help with face to face systems
Needs much discussion.