Transitioning to e-marking to maximise operational

efficiencies: a Caribbean Examinations Council case

study

Dr Anthony Haynes, Assistant Registrar Examinations Development and Production Division, Caribbean Examinations Council (CXC)

Brendon Kavanagh, Head of International Business Development, RM Results

The Caribbean Examinations Council (CXC)

▪ CXC was established in 1972 by participating governments ▪ We operate from two offices based in Barbados (examination

development and administration) and Jamaica (syllabus development) ▪ We service 19 territories in the Caribbean ▪ Our offering:

▪ Internationally recognised examinations and certification ▪ Curriculum & syllabus development ▪ Training & professional development ▪ Consultancy

▪ Our qualifications are recognised in both the UK and the USA ▪ We are proud to serve our island people and diaspora ▪ For more information visit www.cxc.org

Examinations in the Caribbean

Examinations in the Caribbean

CXC provides examinations and certification at five main levels

● CPEA – Caribbean Primary Exit Examination (11 year olds) transitioning from the primary to secondary level

● CCSLC – Caribbean Certificate of Secondary Level Competence - mid secondary level (12-14 year olds)

● CVQ – Caribbean Vocational Qualification (Competence based assessment, mainly technical vocational subjects)

● CSEC – Caribbean Secondary Education Certificate (16 year olds) students transitioning from Secondary school to College

● CAPE – Caribbean Advanced Proficiency Examination (18 year old) students transitioning to college and university

Why Consider E-Marking?

The current process was:

▪ Expensive to run

▪ Cumbersome

▪ Not scalable (we could not do new things)

▪ Resource hungry

▪ Outdated

▪ Restrictive

▪ Low tech – NO tech!

▪ The process consumed the budget

▪ We wanted efficiency and flexibility

Why E-Marking?

E-marking offers cost benefits: ▪ The current CXC model is not sustainable

▪ Scripts marked on paper in centres on different islands

▪ Markers being flown and accommodated to be together

▪ Removes logistics costs - markers can mark from home (or anywhere)

▪ Removes personal costs - time and money

▪ Markers who leave the Caribbean can still mark

▪ Marking quality and value is easier to track

▪ Less errors call for reduced need for re-marking and a heightened confidence in exams.

▪ Reporting on markers and marking behaviours is much quicker and effective

Why E-Marking?

E-Marking offers accuracy benefits: ▪ Greater accuracy of marking and improved quality control

▪ Easy ‘seeding’ process to give markers feedback and markers not meeting standards on seed scripts can be suspended and given additional training

▪ Minimizes potential for script loss or damage

▪ Improved speed and ease of score transmission

▪ Improved real-time management data and reporting

▪ Improved post-exam data analysis

▪ Completely anonymised marking and no halo effect

Why E-Marking?

E-marking offers examiner benefits: ▪ Streamlined process

▪ Fewer administration tasks

▪ Focus on accurate and efficient marking

▪ Real time quality assurance and feedback

▪ Less need for costly and inconvenient travel

▪ Faster rate of marking whilst maintaining quality

▪ Can monitor marker performance on an individual and group level

▪ Displays CXC’s commitment to innovation

Ensuring Quality – The Pilot

▪ CXC have been evaluating potential e-marking systems since 2009 ▪ Partner choice based on quality of e-marking solution first, not

providers ▪ Quality assurance is paramount ▪ Equity of certification must not be compromised ▪ If “face to face” is best, how does E-Marking compare?

▪ Three systems were piloted ▪ RM Results’ e-marking solution, RM Assessor (formerly ‘Scoris Assessor’) was

selected for a larger pilot

▪ In 2013 CXC piloted RM Assessor to mark two subjects - Integrated Science and Physics

▪ Subjects were marked using traditional pen and paper methods along with e-marking to directly compare mark correlation

▪ The results were compelling….

E-marking v traditional pen and paper

High correlation provided CXC with the knowledge that e-marking was as accurate as pen and paper marking

Strength of results provided them with the confidence to increase script numbers in 2014

How to Chose an E-marking Provider?

An established provider of E-Marking was required ▪ High standards of CXC must be met

▪ The pilot must work seamlessly to increase confidence

▪ The solution must be robust and scalable

▪ The local needs of CXC and it’s marking community must be understood and respected

▪ CXC must be able to become self supporting with own helpdesk facilities

▪ The provider must be trustworthy and flexible

The chosen provider – RM Results

www.rmresults.co.uk

E-Marking with RM Assessor

▪ RM Assessor (formerly Scoris Assessor) is the world’s most widely used onscreen marking application for high stakes examinations

▪ In 2013, 143 million scripts in 76 countries were marked using RM Assessor. This figure is set to rise for 2014.

▪ Online application, so no local install is required. Marking tools are clear and intuitive.

▪ Auto-mark adding and online upload, no need to package and distribute physical copies of marked scripts

▪ Enables robust quality assurance processes including standardisation, seeding and double marking

▪ Accommodates a wide range of assessment types that can be marked onscreen

Our clients include:

The CXC E-marking Implementation Plan

9th May First PSD

15th May First Papers ship from Barbados to

Cintas US

12th June Standardisation starts at CXC

16th June Live Marking starts at CXC

28th July Last Marks Upload

9th August Results Issue

19th May First Packet

Arrival at Cintas

11th July Last day of scanning at

Cintas

12th May Packet Arrival at CXC HQ

20th May Scanning begins

at Cintas US

CXC Summer Session 2014

▪ In the May/June 2014 session, CXC examiners used RM Assessor to mark 11 papers across 7 Caribbean Secondary Education subjects: ▪ Biology

▪ Chemistry

▪ Physics

▪ Integrated Science

▪ Principles of Accounts

▪ Principles of Business

▪ Papers were e-marked over a four week period from June - July 2014

CXC Summer 2014 Outcomes

▪ Targets met: ▪ 103,079 scripts (112,753)

▪ 2,290,898 images

▪ 11 components

▪ 664 markers

▪ Scripts per marker: 155 (average)

▪ Examination results released on time ▪ Met target of 100% of papers marked by 20th July

▪ CXC & RM Results established a great working relationship ▪ The Atlantic was not a barrier

▪ CXC marker helpdesk is fully operational with trained advisors on hand in the Caribbean to answer first and second line queries

▪ Over-all CXCs first summer using e-marking was a huge success

CXC Summer 2014 Reflections

▪ Realising benefits

▪ Future cost savings already evident

▪ RM Assessor was easy to use and gave markers flexibility to work from home

▪ The summer was a huge success with the positive energy of chief examiners translating down to the markers with one asking ‘why CXC hadn’t introduced e-marking before?’

▪ Markers completed their quota with some accepting more papers

▪ The CXC administration team were also able to benefit from the flexibility being able to monitor marking progress any time, from anywhere.

Considerations as E-Marking Within CXC Grows

▪ Bandwidth and computer specification

▪ How is broadband delivered and paid for?

▪ Roles may be the same, but the ways of doing them will change

▪ Supervising is now remote

▪ Supervisors have access to more data on marker behaviour

Looking to the Future with CXC

▪ Preparations are now underway for the second round of CXC exams to use RM Assessor

▪ Target for January 2015 session to use RM Assessor to e-mark 100% of CXCs structured papers (around 200,00 scripts)

▪ By summer 2015 all papers to be e-marked (around 500,00 scripts)

▪ Over the next four years, around four million scripts will be e-marked

▪ Maximising efficiency in CXC by utilising the wealth of data now available

▪ Scanning of scripts to be done locally in the Caribbean to support the local economy

Questions…

Please visit the RM Results stand for a complimentary copy of our report entitled ‘From pens down to results day: maximising examiners’ time to improve marking quality’ from our roundtable debate held in April this year at the Royal Society in London.

www.rmresults.co.uk