EFQUEL Academy 1

Quality in Virtual Schools and Colleges – a view from VISCEDPaul Bacsich (paul.bacsich@sero.co.uk)

VISCED – www.virtualschoolsandcolleges.info

EFQUEL Academy 2

Introduction

EFQUEL Academy 3

Topics

• Historical overview• The Pick&Mix system• Supplementary criteria, local criteria and slices• Scoring, evidence and documentation

EFQUEL Academy 4

Benchmarking and Quality in e-learning: My part of the story

Overview and history

EFQUEL Academy 5

Benchmarking e-learning (UK)

• Foreseen in HEFCE e-learning strategy• Higher Education Academy oversaw it• Four phases – 82 institutions (2005-07)• Justified entry to Pathfinder and Enhancement

programmes - and useful for JISC bids (Curriculum Design etc)

• Was leveraged into refresh of learning and teaching strategy (Leicester etc)

EFQUEL Academy 6

What is benchmarking? (HEFCE)

“[for] individual institutions [to] understand their own positions on e-learning, to set their aspirations and goals for embedding e-learning… to benchmark themselves and their progress against institutions with similar goals, and across the sector [world]”

Can replace e-learning by “distance learning”, “OER”, “podcasting” ,etc

EFQUEL Academy 7

Methodologies: Pick&Mix• First used for Manchester Business School 2005• Has been used in 36 institutions, mainly new universities

and university colleges– Including 4 very diverse institutions in Wales (2007-09)– And 5 in the Distance learning benchmarking Club (UK, Sweden,

Canada) plus two more in UK (2009-12)– And basis for the Critical Success Factors schemes in two EU

projects: Re.ViCa (2007-09: developments fed back into Pick&Mix) and now VISCED (2011-12)

– Also an OER version of the scheme ready for use• Wording slightly revised for schools and colleges

EFQUEL Academy 8

Pick&Mix

Criteria and metrics

EFQUEL Academy 9

Criteria

• Criteria are “statements of practice” which are scored into a number of performance levels from “bad” to “best” to “excellent”

• These statements are in the public domain – to allow analysis, refinement etc (cf.OER)

• Pick&Mix has over 80 criteria…– But typically a set of 20-30 is chosen – this range is based

on analysis from the literature (ABC, BS etc) and experience in many scoring meetings

EFQUEL Academy 10

Pick&Mix Metrics

• Use a 6-point scale (1-6) – ignore 6 if you wish– 5 (cf Likert, MIT90s levels) plus 1 more for “excellence”

• Backed up by continuous metrics where possible• Also contextualised by narrative• The 6 levels are mapped to 4 colours in a “traffic

lights” system (red, amber, olive, green)

EFQUEL Academy 11

Pick&Mix

Three sample criteria

EFQUEL Academy 12

P01 “Adoption” (Rogers) – not used now

1. Innovators only2. Early adopters taking it up3. Early adopters adopted; early majority taking

it up4. Early majority adopted; late majority taking it

up5. All taken up except laggards, who are now

taking it up (or retiring or leaving)6. First wave embedded, second wave under way

(e.g. m-learning after e-learning)

EFQUEL Academy 13

P10 “Training”1. No systematic training for e-learning2. Some systematic training, e.g. in some projects and

departments3. Institution-wide training programme but little

monitoring of attendance or encouragement to go4. Institution-wide training programme, monitored and

incentivised5. All staff trained in VLE use, training appropriate to job

type – and retrained when needed6. Staff increasingly keep themselves up to date in a “just

in time, just for me” fashion except in situations of discontinuous change

EFQUEL Academy 14

P05 “Accessibility” – tough one

1. e-learning material and services is not accessible2. Much e-learning material and most services conform to

minimum standards of accessibility3. Almost all e-learning material and services conform to minimum

standards of accessibility4. All e-learning material and services conform to at least minimum

standards of accessibility, much to higher standards5. e-learning material and services are accessible, and key

components validated by external agencies6. Strong evidence of conformance with letter & spirit of

accessibility in all countries where students study

EFQUEL Academy 15

Pause for questions

EFQUEL Academy 16

Extensions, local criteria and slices

EFQUEL Academy 17

Extensions

• Pick&Mix started with 20 core criteria – but now there are over 80 – and a different core (2.0)

• There is room for more supplementary criteria – and further analyses are always being done to ensure topicality

• Web 2.0 aspects are already covered• But it is mainly the institutions who have kept

and will help to keep the system up to date• And some EU and other projects

EFQUEL Academy 18

Supplementary criteria - examples

• They include:– IT reliability– Market research, competitor research– IPR– Help Desk– Management of student expectations– Student satisfaction

EFQUEL Academy 19

Slices

• As well as benchmarking the whole institution it is wise in large institutions to look at a few “slices”:

• Useful to give a context to scores• Slices need not be organisational or subject-based– Distance learning (is a natural one)– Thematic or dimensional like HR, costs…

EFQUEL Academy 20

Scoring

• Scores are usually in the range 1-5, point 6 is exceptional and cannot be planned for

• Focus not only on best practice but also bad practice and real practice

• Do not try to ensure that your institution scores 5 overall

• 6 sometimes has aspects of 1

EFQUEL Academy 21

Use in Distance Learning (such as Virtual Schools)

details

EFQUEL Academy 22

Distance learning• A variant “mood” of the default Pick&Mix was

developed to benchmark distance learning• This was used in the Distance Learning

Benchmarking Club:– University of Leicester– Thompson Rivers University– Lund University– Royal Swedish Technical Institute (KTH)

• and also at the University of Gotland• Most recently at University of Northampton

EFQUEL Academy 23

Evidence and documentation• Normally institutions want a final report but but the

aim is NOT to recreate the QAA-style SED or DATs –the report is an outcome of the benchmarking process not the start of it

• Have a “file” for each criterion• Some narrative will be needed• Institutions normally group criteria according to their

own L&T strategy or in terms of “owning” departments– We also supply some standard groupings based on MIT90s

but few use these

EFQUEL Academy 24

Documentary support

• There are many reports available – many but not all are public, others are available to those participating in Pick&Mix

• The HE Academy wiki “was” a good source of information and the Academy blog structure “was to be” maintained indefinitely…. Not now

• We store all Pick&Mix key material separately from the HE Academy

• There are now many papers and a bibliography is available

EFQUEL Academy 25

Ongoing work

• Evolving Pick&Mix into a benchmarking and quality scheme for virtual schools:– With specialist help from EFQUEL– And a research team at KU Leuven– And correlation with iNACOL scheme in US

• Version of Pick&Mix ready to benchmark OER uptake (developed with Bieke Schreurs, OUNL)

• Variant moods for accessibility and employability (Wales)

EFQUEL Academy 26

Thank you for listening

email for this work is paul.bacsich@sero.co.uk

Benchmarking E-Learning in UK Universities: Lessons from and for theInternational Context , published in http://www.openpraxis.com/files/Bacsich%20et%20al..pdf

http://www.virtualcampuses.eu/index.php/Bibliography_of_benchmarkingandhttp://www.mendeley.com/groups/1075191/benchmarking-e-learning/