Case Studies on Best Practice inExternal Verification

April 1999

Publication code: D0685

Published by the Scottish Qualifications AuthorityHanover House, 24 Douglas Street, Glasgow G2 7NQ, and Ironmills Road, Dalkeith,Midlothian EH22 1LE

© Scottish Qualifications Authority 1999

ContentsForeword

Introduction 1

Case study 1: Association of Industrial Truck Trainers (AITT) 3Case study 2: City and Guilds, Scotland 6Case study 3: EMTA Awards Ltd 9Case study 4: Scottish Qualifications Authority 12Case study 5: Scottish Qualifications Authority 15Case study 6: Vocational Qualifications in Science, Engineering and Technology

(VQSET)18

SQA contacts 21

ForewordCase Studies on Best Practice in External Verification is one of a series ofpublications on SVQ design and delivery that has been produced in response to therecommendations of the Beaumont Report following the review of 100 NVQs andSVQs. An objective of this review was to ‘highlight good practice and inform thedevelopment of guidance in order to promote improvements in NVQ and SVQdevelopment and assessment processes’.

The publications in the series address recommendations arising from this objective bysetting out the principles that underpin SVQs and NVQs, and by giving guidance onbest practice in assessment and verification, including:

♦ improving the language used in SVQs and NVQs to aid comprehension

♦ reducing unnecessary bureaucracy

♦ enhancing the credibility and consistency of assessment

♦ encouraging a more customer-focused approach to the delivery of SVQs and NVQs

The other titles in the series are:

♦ Implementing SVQs: A Guide for Awarding Bodies

♦ Guidance on Learning and SVQs

♦ Developing an Assessment Strategy for NVQs and SVQs

♦ Developing National Occupational Standards for NVQs and SVQs

♦ Developing National Occupational Standards: Guidance on Tailoring

♦ Developing National Occupational Standards: Guidance on European andInternational Benchmarking

Some of these publications have been published in partnership with the Qualificationsand Curriculum Authority (QCA). This document refers solely to SVQs.

Work continues on improving the way SVQs and NVQs are developed, specified andassessed, but the impact of this work on individual SVQs and NVQs will vary interms of both the extent and the timing of changes. In many cases, the updating ofspecific qualifications will follow re-accreditation and revision cycles, which meansthat whilst the form and content of many SVQs and NVQs will stay the same for sometime to come, a number of SVQs and NVQs will take on changes more quickly.

The system and individual qualifications within it will therefore evolve, rather thanchange wholescale, and improvements will be introduced as and when necessary. It isthe intention of both SQA and QCA that this methodology should minimise disruptionand make it possible to build on the strengths of the existing system.

1

IntroductionThis booklet contains six case studies highlighting best practice in externalverification. The case studies have been produced by the Scottish QualificationsAuthority in response to the Beaumont Review — which expressed concerns aboutsome approaches to external verification by the different awarding bodies for SVQsand NVQs.

The case studies feature five awarding bodies:

♦ Association of Industrial Truck Trainers (AITT)

♦ City and Guilds Scotland

♦ EMTA Awards Ltd (EAL)

♦ Scottish Qualifications Authority (SQA)

♦ Vocational Qualifications in Science, Engineering and Technology (VQSET)(formerly the Chemical, Pharmaceutical and Allied Industries Awarding Body)

The case studies cover a range of external verification activities, and show how theawarding bodies are increasingly looking at new technology to help external verifiersprepare for and carry out visits to centres. They also highlight how important it is forexternal verifiers to network with centres, along with reinforcing the externalverifier’s role in giving development advice. The broad headings of what follows are:

♦ Preparation for the visit

♦ Good practice in conducting external verification visits to centres

♦ Keeping external verifiers up to date with awarding body and sector developments

♦ Innovative ways of using information technology

♦ External verifier networking with centres

♦ Systems verification and support to centres on internal quality assurance

♦ Balancing development visits with external verification activity

The case studies illustrate enhanced practice in external verification — what theyshow exceeds the minimum requirements stated in the accrediting body’s criteria. An important point the case studies show is that in preparing for visits to centres it isbest practice to ensure that external verifiers are aware of the centre’s history and, inparticular, of the issues which were raised at the previous visit. This is important fortwo key reasons. First, it reminds the external verifier of any action points that wereraised with the centre at the last visit. Second, there may be some issues for theexternal verifier to discuss with the awarding body — for example, the centre mayhave asked for information which the external verifier was unable to provide on the day.

2

Increasingly, awarding bodies are seeking the views of external verifiers on thecontent of support materials and to help them identify areas where additional guidancewould be helpful for centres. The use of feedback forms for completion by centres following an external verifier’svisit means that awarding bodies can monitor customer satisfaction with the visit. Itcan also help to inform training and development needs for external verifiers. As the accrediting body for SVQs, SQA does appreciate that awarding bodies havefinite resources, and so long as an awarding body meets our criteria it will continue tobe accredited to offer SVQs. Our views on best practice, however, should not beregarded as expensive enhancements to external verification, but instead should beseen as helping to maintain the credibility and rigour of this key aspect of qualityassurance. Note that external verification may also be referred to as ‘moderation’, and externalverifiers as ‘external moderators’.

3

Case study 1 Awarding body: Association of Industrial Truck Trainers (AITT) Areas of best practice: Preparation for the visit

Good practice in conducting external verification visits tocentres

AITT wants to take quality into the millennium — without quality there is norecognition.

Colin Greenwood, AITT

IntroductionThe AITT is a small awarding body established just under four years ago to awardqualifications in the lift truck and construction plant industry. It has six centres inScotland, with about one hundred candidates doing SVQs.

Until six months ago, Colin Greenwood at AITT confessed the external verifiers hadvery little information to go on when arranging visits to centres. Introducing a new ITsystem has meant a significant difference to the amount of data supplied to theexternal verifiers before they visit the centres.

External verification in practice

Preparation for the visitThe AITT’s new IT system prompts the awarding body into arranging externalverification visits two months before they are due to take place. This gives theawarding body's administration staff time to arrange the visit and let the externalverifier know when it will take place. The external verifier is then supplied withinformation on the previous visit and a set of instructions to look at issues highlightedin the report.

Apart from supplying details of the previous visit, the system enables the awardingbody to collate data on centres’ equal opportunities policies and internal verificationactivity. It can also supply external verifiers with print-outs of:

♦ new candidates enrolled at centres since the last external verification visit

♦ candidates with special needs

♦ assessors’ and internal verifiers’ qualifications

4

To maximise the efficiency of the new IT system, the awarding body has had tore-format the external verification reports so that information is gathered in astructured and uniform way. Colin recognises, however, that the data held on thesystem must be constantly updated if it is to be of any real value to the AITT.

Centres are advised in advance of the visit on what evidence and assessment decisionsthe external verifier will look at during the visit. As part of the external verifier'ssampling strategy, the AITT insists that external verifiers see assessments taking placeand talk to candidates. This information is also entered into the IT system and meansthat when centres request candidates’ certificates, the awarding body can do a quickcheck to see if any of the candidates were interviewed on the last external verificationvisit.

AITT is currently piloting its new IT system with five centres and, if successful, the ITsystem will be introduced to all 78 centres throughout the UK. The main advantage ofthe new system to AITT is that there is a common format for all the data held incentres and the awarding body. This makes downloading the data onto a disk so mucheasier for centres to deal with, and the format also helps the awarding body to interpretand verify the data.

Good practice in conducting external verification visits tocentresArmed with data before going out to centres, external verifiers are now asking to seeevidence of staff qualifications and experience as part of reporting back to theawarding body. Evidence of the centre adhering to its health and safety, and equalopportunities, policies is also requested — to the AITT's concern, external verifiershave found that these policies are gathering dust in centres.

External verifiers also carry out unannounced visits on behalf of the AITT. Thesevisits are broken down into two parts:

♦ 100% check of all records held in a centre

♦ external verifiers’ observation of assessment and internal verification as it takesplace

If external verifiers are concerned about what they find on these visits, they canremove evidence and bring it back to the AITT.

At the end of each visit, the external verifier will inform the centre of when the nextvisit is due and will remind them that the findings of this visit will inform the next.

Keeping external verifiers up to date with awarding body andsector developmentsThe AITT places great store on training its external verifiers, and has employed theservices of a consultant to review the way the awarding body manages itsqualifications. The review will cover how the AITT recruits external verifiers, andcould mean that its recruitment policy will be revisited.

5

The consultant has been asked to identify any additional training needs for externalverifiers. At the moment, all the external verifiers are qualified with D32, D33 andD35 and have been given training in communication and report writing skills.

The AITT currently employs a team of four external verifiers, but hopes to increasethis to six. Each external verifier must attend training sessions four times a year inaddition to meetings with other external verifiers. The AITT uses these meetings toconduct cross sampling exercises amongst external verifiers, and encourages externalverifiers to share practice and problems encountered in centres.

The awarding body has set up regional assessment groups, bringing the externalverifiers into contact with a number of centres at the one time. These groups meetthree times a year and are attended by assessors, internal verifiers and centre co-ordinators. External verifiers are responsible for chairing the meetings — the mainpurpose of which is to promote consistency amongst centres offering thequalifications. Colin regards these meetings as essential in keeping the externalverifiers and the AITT attuned to the needs of its centres as well as giving the externalverifiers another opportunity to boost centres' confidence in the awarding body'sapproach to external verification.

Summary of best practice♦ Effective use of IT to prepare external verifiers for visits to centres. External

verifiers are supplied with information which the awarding body has been able tomanipulate from the system.

♦ External verifiers are using this information to carry out more meaningfulinvestigations at centres. The AITT is also introducing unannounced visits tocentres to look at records in detail as well as observing assessments.

♦ The AITT is reviewing its approach to external verifier recruitment and training.There are regular meetings of external verifiers and training events throughout theyear.

6

Case study 2 Awarding body: City and Guilds, Scotland

Areas of good practice: Preparation for the visit Keeping external verifiers up to date with awarding body

and sector developments Innovative ways of using information technology

City and Guilds has an active policy of reviewing and updating its quality assuranceprocedures. This process is not only applied to continuous improvement ofdocumentation, but also to the employment of new developments in technology. Thispolicy is achieved through close contact with our large team of external verifiers andan internal team of quality systems developers. Recent developments have focused ontwo areas — the visit process and the management of information.

We have developed an additional aid to external verifiers, a visit planner. This aidsthe structure of the day and clarifies the audit objectives. Dedicated software supportsthe external verification process by keeping track of all external verifiers andprovides a valuable database on activity.

Murray Butcher, Head of Quality and Audit

IntroductionCity and Guilds, Scotland is based in Edinburgh and opened in September 1995. Theawarding body offers a wide range of SVQs and services in Scotland. The office keepsclose links with the head office in London, but is able to tailor its service north of theborder to meet Scottish needs.

External verification in practice

Planning for the external verification visitCity and Guilds has recently reviewed its approach to external verification — inparticular how external verification visits are planned and prepared.

This has led to the introduction of a ‘visit planner’, which is sent to all centres by theexternal verifier in advance of the visit. The planner informs centre staff of theprogramme for the visit and who the external verifier will talk to on the day. Theprogramme is designed around the external verifier’s sampling strategy, which in turnis determined by the data the awarding body holds on the centre’s registration andcertification activities, and by guidelines issued by City and Guilds.

7

The visit planner is sent to the centre six weeks before the visit. This is to allow theperson responsible for co-ordinating the visit enough time to contact staff andcandidates and ensure they are available to meet the external verifier. Of course, itmay not always be possible for candidates or staff to be around for the visit, but solong as the centre informs the external verifier of this in advance and can provide areasonable explanation for any absences, the external verifier should be happy to carryon with the visit.

As part of the sampling strategy, the visit planner will also identify the records andevidence which should be available for the visit, such as:

♦ the SVQs and units to be sampled

♦ whose assessments the external verifier would like to observe taking place

♦ what details of claims to certification will be inspected

City and Guilds, like all other awarding bodies, is aware that supplying thisinformation in advance to centres can affect the accuracy of the audit. Spaces in thevisit planner where the external verifier indicates that ‘unspecified’ records andevidence will also be looked at during the visit avoid this problem.

Amongst the benefits of introducing the visit planner is a better formal structure tovisits, and the promotion of consistency amongst external verifiers. It also means thaton any return visits to the centre, the external verifier can check on the progress ofcandidates included in the sample for the last visit.

During the course of the visit, external verifiers will refer to the planner and draw up areport on their findings. The report will say whether the centre needs to take anyaction and, depending on the seriousness of this action, whether a follow-up visit isnecessary in the near future.

The external verifier has to write up the report on the day, agree it with the centre asan accurate account of the findings of the visit, and leave a copy for the centre'srecords (and for action, where necessary). The external verifier keeps a copy of thereport, and also sends a copy to City and Guilds Scotland and its lead verifier for thesector. In this way, City and Guilds and the sector lead verifier are informedimmediately of any serious difficulties, and can also update their centre records.

Awarding body support for external verifiersExternal verifiers’ reports make an important contribution to deciding on the supportmaterials and networks City and Guilds should make available to the externalverification team.

The information contained in reports is not only reviewed to define the training andsupport external verifiers may need to do their jobs effectively, but also helps theawarding body, through its sector lead verifiers or national chief verifiers, to spottrends emerging across the different sectors. External verifiers are advised of thesetrends in an awarding body publication EV Update, which covers issues such asdifficulties with interpretation of standards and problems with quality assurance.

8

These publications are used along with other mechanisms (for example externalverifier networks) to encourage consistency and standardisation across the externalverification teams. City and Guilds also holds annual updates for external verifiers aswell as smaller meetings of external verifiers working in the same sector.

Other publications produced by City and Guilds to support external verifiers tell themhow to complete the visit planner, but a key document is Ensuring Quality, which issent to centres and external verifiers. At the same time, additional notes intended onlyfor external verifiers will be included where necessary in the edition for externalverifiers. These notes could include information on new developments by theaccrediting bodies and changes to policy as a result of these developments.

Innovative use of information technologyThe office in Edinburgh is linked to the City and Guilds national IT network, andgives office staff information on external verification activity and centres across thecountry. The system is on-line — as data is entered it will update other informationheld in the system. For example, when a centre is approved, the system will set upfiles for registration and enrolments.

Office staff can also update the system for centres operating exclusively in Scotland.The external verification reports help to build up centre profiles on the system and tokeep records on external verifier training.

The system is particularly useful at alerting the awarding body to when an externalverification visit to a centre is overdue, and any problems in other parts of the UKwhich may have a knock-on effect in Scotland.

City and Guilds has long term plans for the system which include an on-line E mailsystem for external verifiers which would allow them to send completed reportselectronically. The idea of giving all external verifiers laptops for access to thedatabase is also being investigated.

Summary of best practice♦ City and Guilds has recently reviewed its approach to external verification and has

come up with an improved way to plan and prepare for external verification visits.This allows the external verifier to target individuals and records during the courseof the visit as part of the sampling strategy.

♦ External verifiers are supported in various ways by the awarding body includingnetworks, publications, and by identifying and meeting their training needs.

♦ External verification reports are used to build up centre profiles on the IT systemwhich is also used to record details of external verifier training.

9

Case study 3 Awarding body: EMTA Awards Ltd (EAL)

Areas of best practice: Keeping external verifiers up to date with

awarding body and sector developments Good practice in conducting external

verification visits to centres

The external verifier’s role is critical to the attainment of high quality and requiresthe external verifier to reach fair decisions based on mature judgements.

In order to facilitate this, one of EAL’s primary tasks is to provide full support to itsexternal verifiers by open discussion of the real issues and provision of tools andsystems to do the job effectively.

Ron Law, National Verification Manager, EAL

IntroductionEAL develops and awards NVQs and SVQs across a wide range of engineering skillsand occupational areas, such as mechanical engineering, electronics, motor vehicleand the aerospace industry. It supports these with training materials, guidelines andother services, including a team of advisors backed up by a national network ofresource centres.

External verification in practice

Keeping external verifiers up to date with awarding body andsector developmentsEAL employs a team of seventeen full-time external verifiers and assigns eachexternal verifier to a geographical area. The awarding body hosts bi-monthly meetingsof the external verification team, but unlike most other awarding bodies, EAL invitesofficers from the accrediting bodies and DfEE to attend its meetings.

Items on the agenda for the external verifiers’ meeting will focus on issues/problemsfaced by external verifiers on visits, product development undertaken by EAL andtopics such as core skills. During the course of the meetings, Ron Law at EAL likes toencourage the external verifiers to share experiences in the field so that they can learnfrom each other.

10

As all the external verifiers have been issued with a laptop, EAL uses E-mail asanother way of giving its external verifiers advice and guidance when problems arise.However, Ron places just as much emphasis on the external verifiers keeping theawarding body abreast of issues in the sectors and he looks for a monthly report fromeach external verifier to make sure EAL is in touch with what is happening in centres.

Good practice in conducting the external verification visits tocentresPrior to making visits, the external verifiers will use their laptop PCs to accessawarding body information on previous visits to centres and action plans whereappropriate.

Like other awarding bodies, EAL has issued its external verifiers with guidance onhow to set an appropriate sampling strategy. The awarding body has made a consciousdecision to bias the sample to target perceived weaknesses in centres, such as thenumber of new assessors and the centre’s experience of delivering SVQs.

Each centre is visited by an external verifier twice a year, but the number of visits canincrease if the external verifier has concerns about the centre.

Arranging the visit, the external verifier will contact the centre co-ordinator to ask thatstaff and/or candidates are available on the day. The external verifier can be fairlyinsistent about this, but must also be pragmatic if there is a good reason why thesepeople can’t be around for the visit.

The tone of the visit is geared towards improvement in centres — helping them toachieve the quality standards expected by EAL. There are occasions when externalverifiers are asked to give advice on matters which may be outwith their remit, but theexternal verifiers are encouraged to help when they can and to refer issues back to theawarding body when they can’t resolve the queries. One such query on simulation ledto a meeting with one of the accrediting bodies.

Ron is anxious to avoid external verifiers being too heavy handed, and externalverifiers are aware of when it is important to list action points for a centre fallingbelow expectations. These expectations are defined as a series of prompts in thereports used by external verifiers during visits.

To make the report as easy as possible to complete, EAL has designed the form withboxes in which the external verifiers simply insert ‘yes’ or ‘no’ to indicate whether thecentre is continuing to meet requirements. The report form has been loaded onto theexternal verifiers’ laptops, but a paper-based version is also available. The externalverifiers use the report like a checklist, prompting them to look for things during thecourse of the visit. As it is only a ‘no’ entry in a box which raises an action point of aserious nature for the centre, the design of the report is seen by Ron and the externalverification team as helping to concentrate the mind on important issues.

EAL has devised a standard list of action points which an external verifier can refer toif necessary. For example, the external verifier can indicate to a centre when a moreappropriate assessor needs to be found. There is also space in the report form forincluding any additional comments required.

11

At the end of the visit, the external verifier will print off the report to be signed by thecentre co-ordinator and the external verifier. Each external verifier has a portableprinter, but if a copy can’t be left on the day of the visit, EAL requires externalverifiers to send off the report within the week.

Centres are given the opportunity to give feedback to EAL after the external verifier’svisit. There is space in the printed external verification questionnaire for centres tomake their comments.

One major benefit to EAL in giving external verifiers the laptops has been the wayinformation is forwarded to the awarding body immediately after the visit. Theexternal verifiers simply need to plug into a telephone line and then they can updatethe awarding body's system quickly and cost-effectively. The introduction of a reportproforma on the system also means that EAL can gather information from its externalverifiers in the standard way.

Summary of best practice♦ Bi-monthly meetings of the external verification team to keep them abreast of

developments and to give them an opportunity to air concerns/views. EAL invitesofficers from the accrediting bodies and DfEE to attend these meetings.

♦ Innovative use of laptops to communicate with external verifiers when problemsemerge and to help them conduct the external verification visit.

♦ The tone of the external verification visit is based around EAL’s expectations forcentres. EAL’s requirements are defined in the external verification reportproformas, which act as a checklist or series of prompts to the external verifiers asthey carry out their visits. The report has been designed for ease of completion, butit also makes sure that centres are aware of any action points which they need toaddress as a result of an external verification visit.

12

Case study 4 Awarding body: Scottish Qualifications Authority (SQA) Area of best practice: External verifier networking with centres

Quality Networks provide excellent opportunities for SQA to have a ‘finger on thepulse’ — centres are always able to give quick feedback on issues and generally tellus what they are thinking. A particularly satisfying aspect is the sense of workingtogether — this is really at the heart of quality networking.

Patricia McCarron, Approvals Manager

IntroductionSQA is the statutory body in Scotland for developing, awarding and accrediting arange of qualifications for people from school students through to employers. SQAwas established in April 1997 as a result of a merger of the Scottish ExaminationBoard and the Scottish Vocational Education Council (SCOTVEC). It awards a rangeof SVQs on its own or in partnership with other bodies.

External verification in practice

External verifier networking with centresIn March 1995, SQA (then SCOTVEC) set up Quality Networks which bring togethercentres offering SVQs in the same occupational area to share best practice andexperience of delivering and assessing the qualifications.

The networks are regarded by SQA as a way of giving more support to a greater rangeof centres in the approval and verification processes. At one meeting, SQA is able totackle a range of issues common to a variety of centres and give guidance in thecontext of the qualification area.

So far, Quality Networks have been set up for fifteen occupational areas includingManagement, IT, Retail, and Customer Service. At each meeting of the network, SQAaims to get representatives from a spread of centres in terms of the type oforganisation and their experience of offering the SVQs. This means experiencedcentres can share views on how to deal with some of the challenges of implementingSVQs, and new centres can bring a fresh look and ideas on how its staff intend tooffer the qualifications.

13

The Quality Networks are only open to staff from existing approved centres. Thosecentres which are not yet approved, but would like to offer SQA qualifications, attenda different seminar specifically designed to help them meet the approval requirements.Once these centres are approved, they can join the Quality Networks.

Before each network, office staff and the external verifier responsible for running theevent discuss the agenda and identify the support materials required for the day. Formost meetings of the networks, the agenda will be based on requests from centres tocover particular topics. These requests are identified from feedback forms inserted inthe delegate packs issued for each event. By drawing up the agenda in this way, SQAis trying to encourage centres to indicate where they would most like guidance andsupport. As the networks are run to reinforce the message that there is a learning curveto implementing SVQs, centres are encouraged to be open in highlighting issues, nomatter how trivial they may seem.

Some occasions are slightly different, though, with SQA setting the agenda to dealwith common problems identified by the external verifier on visits to centres, orthrough meetings with other external verifiers. These could be difficulties ininterpreting standards, in using simulation as a source of evidence, or simply how tocomplete paperwork.

Other items are added to the agenda to inform centres of developments by SQA orwithin the sector. For example, at the last meeting of the Management QualityNetwork, SQA asked for delegates’ views on the proposal to introduce independentassessment for SVQs. At future Quality Networks, SQA will also have to talk tocentres about changes to external verification and what impact this will have on them.This is as a result of the merger of SQA’s two predecessor organisations and the needto align approaches.

Patricia and her colleagues look on the Quality Networks as a good opportunity forexternal verifiers to update centres at the same time, promoting consistency and bestpractice in assessment decisions across the centres. This message is reinforced in thelatest edition of the newsletter to external verifiers, In the Picture, which features anarticle on the Quality Networks. The article describes how the networks can be usedas a vehicle by the external verifiers to tell centres about their concerns, as well asexplaining the support external verifiers receive in terms of planning meetings for thenetworks and preparing support materials.

But what do the external verifiers get out of attending and running the QualityNetworks? According to Donnie Carthew, SQA’s full-time external verifier inAdministration, the networks are an opportunity for the external verifier to interactwith representatives from all centres at the one time.

Donnie admits that when he first took on the role of full-time external verifier inAdministration, he had concerns about running a network and how to make the dayvaluable to centres. It was during preparations for his first network that he realisedthat, whilst he had a role to play before and during the network, the most importantpeople are the delegates from the centres and the issues they want to discuss.

14

A typical agenda for a Quality Network in Administration includes a short input fromthe external verifier highlighting new issues and reporting back on the findings fromvisits undertaken by other external verifiers in Administration. The agenda usuallyincludes an item by a ‘guest speaker’ highlighting wider SQA activity, such asdevelopments in Modern Apprenticeships. The rest of the day is given over toworkshops on looking at candidate evidence and portfolio presentation, with regularinputs from the external verifier to help delegates focus their discussions.

Donnie thinks that his role after the Quality Network can be just as important inensuring the quality of the SVQs. He makes a point of telling other external verifiersfor Administration about the issues discussed by the delegates at the Quality Networkor raised in the feedback forms. From an external verifier’s point of view, the QualityNetworks are a means of improving knowledge, consistency, experience, andassessment skills in SVQs.

Summary of best practice♦ SQA has given external verifiers an opportunity to meet with a number of centres

at the same time. This has allowed external verifiers to deal with difficultiesidentified on visits or through meetings with other external verifiers in the samearea, as well as informing delegates of SQA’s latest developments.

♦ This case study also highlighted some of the support available to external verifiersfrom the SQA in the form of opportunities for external verifiers to meet togetherand an external verifiers’ newsletter to keep the team informed of developments.

♦ Delegates have an opportunity to network with other centres and are encouraged toraise queries with external verifiers in an informal setting. It is often the case thatdifficulties experienced in one centre are common to another and attendance at theQuality Networks is another opportunity to resolve problems.

15

Case study 5 Awarding body: Scottish Qualifications Authority (SQA) Area of best practice: Systems verification and support to centres on internal

quality assurance Balancing development visits with external verification

activity

By adopting many of the same strategies for standardisation that we encourage ourcentres to use, the systems verifier team ensures a consistent approach to themonitoring of approval criteria, and the development of our customers’ qualityassurance systems.

SQA Systems Verifier

IntroductionUnder SCOTVEC, one of SQA’s predecessor organisations, the role of the systemsverifier was set up to ensure that qualifications are subject to internal qualityassurance within centres. A systems verifier is not a subject specialist, but insteadgives specialist advice on how to meet, and continue to meet, SQA’s requirements forcentre approval.

SQA has a team of systems verifiers deployed geographically. This team carries outdevelopment visits as well as ongoing monitoring visits to centres.

External verification in practice

Systems verification and support to centres on internalquality assuranceHaving separated out the functions of external verification of subject areas from theverification of a centre’s internal quality assurance arrangements, it is important toSQA that there is communication between the teams of external verifiers (subjectspecialists) and systems verifiers.

Systems verifiers are deployed by SQA to establish that, where there is evidence of aproblem in a centre, it is not an indication of more serious difficulties with the centre'sinternal quality assurance. Where an external verifier has expressed concerns aboutthe effectiveness of a centre’s internal verification, or feels that the centre wouldbenefit from a development visit by a systems verifier, this could prompt adevelopment visit by a systems verifier. The systems verifier will decide if the visitshould take the form of a development visit or a verification visit, depending on the

16

nature of the external verifier’s concern. The purpose of the visit is to look, not only atthe specific problem with the SVQ, but also to verify that other aspects of the centre’sinternal quality assurance arrangements are operating effectively.

For example, the systems verification visit may show that it is not the actual internalverification system which is the problem, but that the staff undertaking the internalverification roles need further training. In this case, it is the centre’s selection,recruitment, appraisal, and development procedures which lie at the root of theproblem, and the systems verifier will want to be assured that this is not having anadverse effect on other areas of SQA provision.

Balancing development visits with external verificationactivityOne of the key roles of the systems verifier is to share best practice. In response torequests from centres, the systems verifier team has produced flowcharts on:

♦ how to track candidate progress

♦ the administration procedures for registration and enrolment for SQA qualifications

♦ how to prepare for an external verification visit

The main benefit to centres is that their staff don’t have to struggle with developing anew procedure or set of materials when SQA can give it to them ‘off the shelf’. ToSQA, the flowcharts are an opportunity to show centres what we require of internalquality assurance, whilst demonstrating to centres that such arrangements do not needcomplex procedures and paperwork to support them.

Another development undertaken by systems verifiers in response to requests fromcentres is the drawing up of job descriptions for the roles of adviser, assessor, andinternal verifier for SVQs. The job descriptions have been defined using the D-units,and include information on what training should be given to support these roles, and alist of useful publications available from SQA.

These job descriptions will be particularly useful to new centres thinking about staffrecruitment and selection. For existing centres, the job descriptions could be used as abasis for staff appraisal and development. Like the flowcharts, the job descriptionsmean centres don’t have to worry about finding the resources to produce thesematerials, and for SQA, they give a clear message to centres on what these rolesinvolve.

The flowcharts and job descriptions were developed during the monthly meetings ofthe systems verifier team. The meetings are seen as a two-way flow of informationbetween systems verifiers and SQA. Typical items on the agenda include issuesarising from visits to centres, such as credit transfer from the Scottish QualityManagement System (SQMS) and views on how to tackle difficulties encountered onvisits. For SQA, the meetings are a chance to inform systems verifiers ondevelopments and to seek their views.

17

One key strength of the systems verifier team is the degree of consistency among themembers of the team. This has been achieved by a planned programme of workshopactivities and regular shadowing of each other during visits to centres. This ensuresthat advice given by members of the team is consistent. The workshops have includedtopics such as how to interpret the approval criteria, report writing, and developingreporting mechanisms.

The systems verifiers have also looked at the report used on systems verification visitsand devised a checklist against which they can check that the content of the report hasbeen met. This helps to standardise the approach taken by the systems verifiers onvisits to centres.

As happens after external verification visits, centres are given feedback forms tocomplete after a systems verification visit. The purpose of these forms is to gatherinformation from centres on the usefulness of the visits and to assist in identifyingsystems verifiers’ training needs.

Summary of best practice♦ Opportunities for the systems verifier team to respond to external verifiers’

concerns about internal quality assurance in centres.

♦ Regular meetings of the systems verifier team to encourage consistencybetween the systems verifiers by discussing any problems encountered onvisits and ways of resolving them.

♦ Developments undertaken by the systems verifier team in response to requestsfrom centres for advice and guidance. Such developments bring benefits tocentres and SQA.

18

Case study 6 Awarding body: Vocational Qualifications in Science, Engineering and

Technology (VQSET) Area of good practice: Keeping external verifiers up to date with awarding

body and sector developments Good practice in conducting external verification visits

to centres

External verifiers play an enormous part in the awarding body. They don't justexternally verify centres. They influence what we do and contribute to developments.They can do this because they understand the needs of our centres. By involving theexternal verifiers in our decisions, they take ownership of our awards and work hardto make them a success.

James Holyfield, Operations Manager, VQSET

IntroductionVQSET is the new name for the awarding body formerly known as the Chemical,Pharmaceutical and Allied Industries Awarding Body.

The awarding body was first established in 1993 and offers NVQs and SVQsthroughout the United Kingdom in:

♦ Engineering Maintenance

♦ Laboratory Operations

♦ Process Manufacture

♦ Process Operations VQSET plans to offer additional NVQs and SVQs in packaging, laboratory operationsfor technicians working in schools and colleges, materials handling, chemicalengineering and analytical chemistry.

External verification in practice

Keeping external verifiers up to date with awarding body andsector developmentsVQSET has recently reviewed its approach to external verification, resulting in theawarding body employing a team of four full-time external verifiers. Each externalverifier makes approval and verification visits to centres within a geographical areaand, according to James Holyfield at VQSET, this has led to improved liaisonbetween the centres and the external verifiers.

19

James encourages the external verifiers to build good working relationships withcentres and to promote centre networks through meetings and phone calls — anactivity which the Beaumont Review highlighted as an area of good practice byawarding bodies.

James is keen to make sure, however, that there is a two way flow of informationbetween the awarding body and external verifiers. To this end, he meets with theexternal verifiers regularly to update them on VQSET's product development and onany policy changes being proposed by QCA and SQA. Most of the meeting is takenup with the external verifiers discussing concerns arising from visits to centres andbest practice they have come across. The external verifiers then identify areas wherefurther guidance to centres is needed, so that speedy resolutions can be provided.

Best practice in assessment and quality assurance features prominently in theawarding body's guides to centres. VQSET issues a range of support materials such asGuidance Notes on Good Practice. The latest publication has just looked at portfoliobuilding and models of delivery for SVQs and NVQs. Other support materials includegeneric guides on internal verification, and the process of external verification isdescribed in a new User Guide.

In producing the guides, VQSET seeks advice from its external verifiers on content.This involvement in support materials not only encourages the external verifiers tofeel a greater sense of ownership over the awarding body's publications, but can alsoindicate to VQSET when it would be worthwhile to involve centres in developments.Centres, for example, could provide material or ideas worth sharing with otherorganisations.

Good practice in conducting external verification visits tocentresVQSET is acutely aware of the importance of getting the balance right betweendevelopment and verification visits. James does not regard the external verifiers asresponsible for policing the system, but sees them as responsible for confirming thatcandidate evidence and assessment decisions are in line with national standards.

In VQSET’s experience, employers want to raise standards just as much as theawarding body does, and they look to the external verifiers for guidance on how toachieve this aim. In real terms, it means that the awarding body has experienced littlein the way of difficulty with centres. External verifiers are discouraged from being tooprescriptive in the advice they give to centres and instead are asked to give broadguidance on how centres can meet VQSET’s requirements and their own businessobjectives — for example, external verifiers might provide example of systems forcandidate induction.

During the course of visits to centres, the external verifiers will take a close look atrecord-keeping. This is given a higher priority when the centre is requestingcertification for the first time. VQSET does not award certificates to candidates at newcentres until external verifiers have conducted monitoring visits and confirmed thatthe centre is complying with the awarding body's requirements for certification.

20

External verifiers are encouraged to minimalise their form-filling, but, where a centrehas been left action points following a visit, the centre must report progress on howthe actions are being met.

One approach to implementing SVQs and NVQs which the external verifiers havefound most effective is encouraging centres to carry out a task analysis of keyactivities and job roles before they look at the standards. This means that assessorsand candidates can see how the tasks undertaken by candidates meet the standards. Atthe end of the matching, the candidates know that in carrying out these tasks, they aremeeting the standards.

As a result of doing the matching exercise, employers have reviewed their proceduresto make sure they are line with the national standards. As a result, existing procedureshave been improved, and new procedures have been developed where they did notexist before.

At the end of the visit, the external verifier discusses the report with the centre co-ordinator and forwards them a copy as soon as possible.

Summary of best practice Regular networking between the awarding body and the external verifiers, including:

♦ external verifiers are involved in producing support materials

♦ striking the balance between development visits and external verification visits,including external verifier guidance to centres on how to get started with SVQs andNVQs

21

SQA contactsIf you would like more information, please contact the SQA Accreditation Unit:

Epic House28-32 Cadogan StreetGlasgowG2 7LP

Tel: 0141-226 4355Fax: 0141-204 5730E-mail: mail@sqa.org.uk