Embed Size (px)
Citation preview
Netherlands Quality Agency
Self-Assessment Report
ENQA Review
Netherlands Quality Agency (NQA)
Catharijnesingel 56-6
P.O. Box 8240
NL – 3503 RE Utrecht
© Netherlands Quality Agency (NQA)
November 2017
2
Table of contents Page
Introduction 5
Development of the SAR 6
1. Dutch system of higher education 7
1.1. Higher education in the Netherlands 7
1.2. Universities of Applied Sciences 8
1.3. Higher Education in numbers 8
2. System of quality assurance higher education 10
2.1. Historic context of the system 10
2.2. Assessment framework 11
3. Netherlands Quality Agency 13
3.1. Brief NQA history 13
3.2. SWOT analysis 14
4. NQA’s higher education quality assurance activities 15
4.1. Assessments of degree programmes 15
4.2. Research evaluations 16
4.3. EVC audits 17
4.4. International degree programme assessments 18
4.5. NQA hallmark for study programmes 18
4.6. Certification of internal audit processes 18
4.7. Other NQA activities 18
5. NQA’s quality assurance processes and their methodologies 20
5.1. Assessments of degree programmes 20
5.2. Research evaluations 22
5.3. EVC audits 25
5.4. International degree programme assessments 26
5.5. NQA Hallmark for study programmes & Certification of internal audit
processes
26
6. NQA - Internal Quality Assurance 27
7. ESG Part 3: Compliance chapter 29
7.1. Activities, policy and processes for quality assurance (ESG 3.1) 29
7.2. Official status (ESG 3.2) 31
7.3. Independence (ESG 3.3) 33
7.4. Thematic analysis (ESG 3.4) 34
7.5. Resources (ESG 3.5) 36
7.6. Internal quality assurance and professional conduct (ESG 3.6) 37
7.7. Cyclical external review of agencies (ESG 3.7) 39
8. ESG Part 2: Compliance chapter 40
8.1. Consideration of internal quality assurance (ESG 2.1) 40
8.2. Designing methodologies fit for purpose (ESG 2.2) 43
8.3. Implementing processes (ESG 2.3) 45
8.4. Peer-review experts (ESG 2.4) 47
8.5. Criteria for outcomes (ESG 2.5) 49
3
8.6. Reporting (ESG 2.6) 52
8.7. Complaints and appeals (ESG 2.7) 53
9. Information and opinion of stakeholders 56
9.1. Evaluation of degree programme assessments 56
9.2. Account evaluations 57
9.3. Consultations with the NVAO 57
9.4. ISO audit (9001) 57
10. Current challenges and areas for further development 59
4
5
Introduction NQA is a Dutch based quality assurance agency with a clear focus on Universities of
applied Sciences in The Netherlands. It’s solid key objective is to assess degree
programmes of universities of applied sciences. With this ENQA review NQA wants to
show its level of compliance with the ESG standards. Next to the assessments of
degree programmes, NQA offers several other (evaluative) services. Amongst others,
this includes research evaluations and EVC audits.
In the first two chapters of this SAR the Dutch system of higher education (chapter 1)
and the system of external quality assurance (chapter 2) are addressed. The following
four chapters focus on NQA starting with a brief introduction, including a SWOT-
analysis (chapter 3), an overview of the NQA’s higher education quality assurance
activities (chapter 4), its quality assurance processes and their methodologies
(chapter 5) and in chapter 6 it’s a description of NQA’s internal quality assurance.
After these chapters, NQA shows its level compliance with the ESG standards. Chapter
7 concerns ESG Part 3, followed by ESG Part 2 in chapter 8. The SAR concludes with
two chapters that provides information and opinions of stakeholders (chapter 9) and a
chapter on current challenges and areas for further development.
6
Development of the self-assessment report (SAR)
For the develoment of the self-assessment report a team of two staff members was
appointed. These two project managers started with an orientation on the European
Standards and Guidelines and talked with a Dutch ENQA member to obtain more
insight in the procedure and their experiences of the review process.
The self-assessment report follows the structure of ENQA’s Guidelines for ENQA
Agency Reviews (2016). As one of the two team members appointed for the
development of the self-assessment report was on maternity leave, the other staff
member took on a larger part in the writing process. In two feedback-rounds, the NQA
director and the NQA staff members shared their input on a draft version of the
report. As the content of the self-assessment report relies strongly on the processes
and methodologies the staff members developed for our activities, their input is also
embedded in that way. This regards for example NQA’s Guidebook for assessments in
higher edcuation (updated yearly).
Next to this document which in detail describes our methods of working regarding the
assessments of degree programmes, internal policy papers as well as external
documents such as the official assessment framenworks are used. Also, we included
the results of several evaluations. The NQA director checked the final draft version,
before sending it to ENQA. This current version of the self-assessment report takes
into account the feedback from ENQA’s reviews manager.
7
1. Dutch System of Higher Education 1.1 Higher education in the Netherlands
Higher education in the Netherlands is offered at two types of institutions: research
universities and universities for applied sciences. Research universities include general
universities, universities specialising in engineering and agriculture, and the Open
University. Universities of applied sciences include general institutions as well as
institutions in a specific field such as agriculture, fine and performing arts or teacher-
training. Whereas research universities are primarily responsible for offering research-
oriented programmes, universities of applied sciences are primarily responsible for
offering programmes of higher professional education, which prepare student for
specific professions. These tend to be more practice oriented than programmes
offered by research universities (EP Nuffic, 2016).
The figure below shows the structure of the Dutch educational system. The position of
the universities (both: research and of applied sciences) is shown in this figure. It
shows the three cycles within the higher education system (bachelor, master and
PhD) as well as the requirements for access to higher education.
8
1.2 Universities of Applied Sciences
As universities of applied sciences are the main clientele of NQA, it’s worth
highlighting a few specifics of these universities. As being said, higher professional
education is primarily offered by the universities of applied sciences. Within the higher
professional education are seven sectors: agro & food, economics, health care,
education, social studies, arts and bèta studies. Within these sectors, students can
choose various educational profiles.
Higher professional education is divided into two cycles. The first cycle lasts four years
(240 EC) and students are awarded a bachelor degree (EQF level 6). The study
programme in the first cycle is divided into a propedeuse (the first year) followed by a
main programme of three years. The fourth year includes a final paper and/or a
graduation project. A short cycle exists within the first cycle: the associate degree
(AD). These AD-programmes last two years (120 EC). After obtaining an associate
degree, students may choose whether to enter the labour market or to follow a
corresponding bachelor programme.
The second cycle in higher professional education usually lasts one to two years. This
cycle leads to the professional master degree (EQF level 7). During the master
programme, the students specializes further in carrying out a particular profession. A
final paper and/or graduation project forms part of the programme, with the emphasis
on applied research.
1.3 Higher education in numbers
To provide more insight in the higher education landscape, the below situated table
shows the number of institutions (funded) and the amount of students. As NQA
focusses its activities on universities of applied sciences, a further elaboration of these
numbers is provided1.
Number of institutions (government funded)
Number of students
Universities of Applied Science 37 446.638
Research Universities 13 264.838
1 http://cijfers.vereniginghogescholen.nl/index.htm and http://www.vsnu.nl/f_c_studenten_downloads.html
9
Master Bachelor Associate degree
Number of students universities of applied
sciences (2016)
11.976 427.869 6.793
Number of degree programmes per sector & number of students
Agro & Food 1 (76) 22 (11.192) 12 (299)
Bèta 5 (71) 42 (87.054) 24 (895)
Economics 1 (69) 52 (168.459) 29 (4.379)
Health 5 (1.299) 22 (45.291) 3 (281)
Arts 32 (2.870) 17 (16.837) 4 (145)
Education 25 (6.603) 32 (48.546) 12 (275)
Social studies 4 (988) 13 (50.490) 4 (519)
As programmes can be offered by different universities, the total amount of degree
programmes is much higher. And it has to be noted that unfunded programmes are
not included in these numbers, while these can also be accredited. For a more
complete overview the visitation schedule2 is a helpful instrument. The schedule
shows that there are about 1.316 accredited degree programmes offered by
universities of applied sciences.
2 https://www.nvao.net/actueel/publicaties/visitatierooster
10
2. System of quality assurance higher education 2.1 Historic context of the system
Before 2002 assessments of degree programmes within higher education were
conducted by the Association of Universities of Applied Sciences (in Dutch: the HBO-
raad, now called the Vereniging Hogescholen) and the Association of Research
Universities in the Netherlands (in Dutch: the VSNU or Vereniging van
samenwerkende Nederlandse universiteiten). Supported by staff members of the
associations, the assessment were carried out by panel of experts (peers) who were
appointed by the associations. The focus of the assessments was twofold: (1) to verify
whether the degree programmes met generic quality standards laid down in an
assessment framework (accountability) and (2) to provide advice and
recommendations for further improvement of the programmes (quality improvement).
In 2002, the Act on Higher Education and Scientific Research was amended to include
accreditation. In the Act, accreditation is described as a quality mark which expresses
that the quality of a degree programme has been assessed positively. Accreditation is
a condition for receiving public funding and for the right to award legally recognized
degrees. In effect, the NAO (the Dutch Accreditation Organisation) was established.
The NAO received the legitimate power to award accreditation to programmes which
fulfilled the conditions laid down in the amended Act. After a short period of time, a
decision was made to combine the accreditation of Dutch education and education in
Flanders (the Dutch-speaking part of Belgium). From then on, the NAO became the
NVAO, the Dutch-Flemish Accreditation Organisation (in Dutch: Nederlands-Vlaamse
Accreditatie Organisatie). The NVAO is responsible for the development of
accreditation assessment frameworks that serve as a starting point for assessments in
the Netherlands and Flanders.
Another consequence of the Act, was that external quality assurance was no longer
considered to be the responsibility of the higher education sector itself. The quality
assurance departments of above mentioned associations were privatized into
independent organisations that continued the assessment activities of the
associations. NQA, founded in 2003, originates from the Netherlands Association of
Universities of Applied sciences. QANU, another quality assurance agency in the
Netherlands, emerged from the association of research universities.
This accreditation system, which can be considered a two-tier system, is nowadays
still in place. The NVAO decides whether an existing programme is (re-)accredited, but
it does not conduct the assessments of the existing degree programmes itself. The
assessments are carried by quality assurance agencies, at the request of the
institutions.
11
2.2 Assessment framework
2003-2011: Start of accreditation
The first NVAO assessment framework for accreditation dates from 2003. This
framework contained 21 standards that were divided into six themes: (1) Aims and
objectives of the programme, (2) Curriculum, (3) staff, (4) Facilities, (5) Internal
quality assurance and (6) Results. Each of them divided into two or more standards.
All degree programmes were assessed based on this framework. And the accreditation
procedure, laid down in the NVAO’s assessment framework, prescribed that
institutions of higher education involved a quality assessment agency to assess their
degree programmes. According to the framework “accreditation is granted after the
NVAO has validated an external assessment, carried out by a quality assessment
agency at the request of the institution”. The duration of accreditation is six years.
2011: Institutional audit, limited and extensive framework
Since the introduction of the two-tier system and the above framework, the external
quality system was revised twice. Since 2011 higher education institutions can request
the NVAO to conduct an institutional audit (optional). The institutional audit assesses
the educational vision of an institution and the policies for putting this into practice,
including human resources and internal quality assurance. A positive result entitles
institutions to use a limited framework for the assessments of its degree programmes.
This limited framework consisted of three standards: (1) intended learning outcomes,
(2) Teaching-learning environment and (3) Assessment and achieved learning
outcomes. If an institution doesn’t apply for an institutional audit, or fails to obtain a
positive result, its degree programmes are assessed on the basis of the extensive
accreditation framework. The extensive framework consisted of sixteen standards,
divided into six themes: (1) intended learning outcomes, (2) curriculum, (3) staff, (4)
services and facilities, (5) quality assurance and (6) assessment and achieved
learning outcomes. Another change in the system regards the initial accreditation of
degree programmes. Since of January 2011, initial assessments are carried out
exclusively by the NVAO.
2014-2015: Intermediate adjustments
In the period 2014-2015 intermediate adjustments were made regarding the
procedure of accreditation as well as to the assessment framework. These
adjustments followed up on several evaluations of the quality assurance system, for
example by the Inspectorate and the Ministry of Education3. In general the outcome
was positive. In a report to the Dutch Parliament, the Minister of Education qualified
3 https://www.onderwijsinspectie.nl/documenten/publicaties/2013/09/12/de-kwaliteit-van-het-nederlandse-
accreditatiestelsel-hoger-onderwijs
12
the Dutch accreditation system as robust and functioning well. The report praised the
expertise of expert panels and the quality of their judgements, just as the quality of
the processes, the flexibility and the capacity of self-improvement of the various
agents in the process. Next to the positive findings, a number of points for
improvement were mentioned. These included ‘the reduction of the administrative
burden’, ‘the improvement of the consistency, reliability and validity of judgements’,
‘more flexibility in instrumentation’, ‘more ownership of quality assurance by teaching
staff and students’ and ‘more trust in the case of proven quality’.
On a system level this led to the introduction of cluster groups and a new visitation
schedule to make this possible. Cluster groups are groups of programmes in a similar
discipline. The most important change in the assessment framework was the division
of the single standard on assessment and achieved learning outcomes (standard 3)
into two separate standards: one concerning (the system of) assessment and the
other concerning the achieved learning outcomes.
2016: Emphasis on trust and ownership
After the 2013 evaluations the Ministry installed an advisory group to elaborate on the
future design of the external quality assurance system. This advisory group presented
its recommendations in a report with the title: ‘Tailor-Made Accreditation’ (in Dutch:
Accreditatie op Maat4). The now prevailing assessment framework is the result of this
initiative. This 2016 framework came into force in 2017, with a transition period until
May 2018. As the NVAO states, the new framework optimizes existing practices and
procedures in order to reduce the administrative burden, to increase the flexibility of
the system and give actors more ownership over assessment procedures.
The 2016 framework is based on the notion of trust and emphasizes the development
of a quality culture. At the same time, it ensures a threshold for basic quality for new
and existing programmes. The NVAO presented the most important differences on its
website5.
All discussed adjustments regarding the accreditation system and the assessment
framework were translated in the NQA work processes. Annex 1 holds the latest NVAO
Accreditation Framework and annex 2 the adoption of this framework by NQA in its
‘Guidebook audit visits in Higher Education, Limited Study Programme Assessment’
(2017).
4 https://www.rijksoverheid.nl/documenten/rapporten/2015/06/02/accreditatie-op-maat (only in Dutch)
5 https://www.nvao.net/system/files/procedures/Important%20differences%20assessment%20frameworks%202014-
2016.pdf
13
3. Netherlands Quality Agency 3.1 Brief NQA history
NQA is a quality assurance agency that focusses on providing services to (mainly)
institutions for higher professional education in the Netherlands. In particular, NQA
conducts assessments of degree programmes on the basis of the formal accreditation
assessment framework.
NQA originates from the Netherlands Association of Universities of Applied Sciences.
In accordance with the amended Act on Higher Education and Scientific Research
(2002), assessments of degree programmes were to be conducted by independent
quality assurance agencies. Subsequently, NQA was formally founded December
2003. NQA is a private organization, since 2008 owned by drs. E. Schalkwijk. The
director, drs. P.T.H. Thijssen, is responsible for the day-to-day management of the
organisation.
After the division from the Association of Universities of Applied Sciences, NQA
invested in standardizing and automatizing its work processes in order to safeguard
an effective and efficient practice. It enabled an assessment approach focused on the
content of the degree programmes and its achievements, rather than on procedures
and processes. Another transition regards the mentality of the staff members. The
association’s monopoly position regarding the assessments of degree programmes for
universities of applied sciences, led to a more ‘civil service-mindset’. As a private
organization and an over the years growing competition between assessment
agencies, a more service-oriented mindset was needed. This service-oriented mindset,
with respect to the boundaries as an independent assessment agency, meant for
example that NQA intensified its role to inform institutions and degree programmes on
the system of accreditation, the framework and the underlying procedures. All to
ensure degree programmes are well-informed on their role and tasks (for example
regarding the self-evaluation).
14
3.2 SWOT-analysis
Strenghts
- NQA is a strong brand within the sector
of universities of applied sciences.
- Its profile on assessments of degree
programmes (focus on content, rather
than on processes) is well-known in the
field and acknowledged by its clients and
the NVAO.
- NQA’s approach and working methods
are in accordance with the relevant
frameworks.
- The staff members of NQA are all well
qualified and equipped for their work.
- Satisfaction of our clients (evaluations
degree programmes and account
evaluations) and the NVAO (expressed in
consultations)
- NQA has an effective set of tools for
safeguarding and improving the quality
of its activities.
Weaknesses
- Despite our strong position as an
assessment agency and our portfolio of
conducted assessments, our influence on
the level of the accreditation system is
limited.
- NQA depends on the visitation schedule
of the NVAO, which leads to peaks and
valleys in our project calendar and thus
on our turnover.
- The frequency of updates of the relevant
systems and frameworks we use as a
starting point for our activities has put
pressure on the organization. The
adoption of these updates in our formats
takes time, which is not always
available.
Opportunities
- Until recently, the assessments of
degree programmes of the research
universities were exclusively conducted
by one assessment agency. Currently it
seems that this sector is a little more
inviting to other assessment agencies.
- Our portfolio of research evaluations is
growing. With this expertise, a (light)
grow of this activity could be achieved.
- Exploring contributions/activities of NQA
that address the quality culture within
institutions and degree programmes and
products that show that degree
programmes are permanently
accreditation-proof.
Threats
- Developments regarding the system of
external quality assurance are difficult to
influence, partly because this is a
political issue. Changes regarding the
position of assessment agencies could
affect us directly.
- It is inherent to our position as an
assessment agency that insufficient
assessment results can influence the
reputation of NQA.
- The ongoing discussion on ‘the
administrative burden’ of the system of
accreditation, can result in a ‘race to the
bottom’ within the system. It is our task
to safeguard a reliable and valid
assessment procedure. We feel the
framework has loosened valuable
elements to ensure this.
15
4. NQA’s higher education quality assurance activities The degree programme assessments are NQA’s main activity. Next to the
assessments NQA conducts several other activities that will be addressed in this
chapter. Providing insight in the activities as well as in its frequency stipulates the
balance within our portfolio.
4.1 Assessments of degree programmes
Accredited degree programmes of higher education institutions need to be assessed
once every six years. In 2014/2015 the NVAO published a visitation schedule (last
update: July 2017) containing the cluster groups and establishing the deadline for
institutions to submit an assessment report. NQA organises and co-ordinates many of
these assessments for universities of applied sciences. For a description on the Dutch
accreditation system, the assessment framework and the NQA assessment procedure,
see chapter 2 (including annexes 1 and 2).
The following table presents an overview of the assessments of degree programmes
carried out by NQA since 2013:
# assessments # degree programmes included in the assessments
Limited versus extensive framework
2017 80 assessments 106 degree programmes: 79 bachelor 25 master
10 AD
84 limited 22 extensive
2016 45 assessments 50 degree programmes: 39 bachelor
9 master 2 AD
45 limited 5 extensive
2015 15 assessments 56 degree programmes: 52 bachelor
4 master
34 limited 22 extensive
2014 67 assessments 81 degree programmes: 54 bachelor 22 master
3 AD
51 limited 30 extensive
2013 105 assessments 122 degree programmes: 98 bachelor 12 master
12 AD
89 limited 33 extensive
Total 312 assessments 415 degree programmes
Overview assessments of degree programmes by NQA, 2013-2017
16
4.2 Research evaluations
Universities of applied sciences conduct applied research activities. Since 2007, the
universities as a sector work together to safeguard the quality of this applied
research. A protocol on quality assurance of applied research first came into practice
in 2009. In 2015 a revised protocol was introduced. Installed by the Association of
Universities of Applied Sciences, a committee validates the quality assurance system
of these universities concerning their applied research: Validation committee quality
assurance research6 (in Dutch: Validatiecommissie Kwaliteitszorg Onderzoek, VKO). A
validation by the VKO is valid for six years. Whereas the VKO assesses on the level of
the institution, the system of quality assurance prescribes that the universities’
research units are assessed by an independent external panel. NQA organises and co-
ordinates these research evaluations.
Over the past five years NQA conducted the following research evaluations:
# Research evaluations
Name of the research unit (translated into English)
2017 4 evaluations - Social (UAS Arnhem & Nijmegen)
- Innovative Entrepreneurship (UAS Rotterdam)
- Sustainable Harbor City (UAS Rotterdam)
- Team Applied Research (UAS Aeres)
2016 2 evaluations - Quality of Learning (UAS Arnhem & Nijmegen)
- Delta Academy (HZ UAS)
2015 3 evaluations - Healthcare Innovation (UAS Rotterdam)
- Biobased Economy (UAS VHL)
- Agribusiness (UAS VHL)
2014 1 evaluation - CAH Vilentum
2013 3 evaluations - Creating 010 (UAS Rotterdam)
- Talent Development (UAS Rotterdam)
- Stoas|Vilentum UAS
Total 13 evaluations
Overview research evaluations by NQA, 2013-2017
6 Webpage VKO (in Dutch)
17
4.3 EVC audits
EVC stands for earlier obtained competencies (in Dutch: eerder verworven
competenties). NQA is one of three agencies appointed by the National Knowledge
Centre EVC (NKC-EVC) allowed to assess EVC providers. Only positive assessed EVC
providers can be registered in the national register of the NKC-EVC, meaning that only
an assessment carried out by one of the three agencies can lead to a registration.
Recognized EVC providers (registered) issue certificates that meet a quality standard
(so called ‘EVC Code 2.07’). These certificates represent the value of the working
experience of a candidate in comparison to the qualifications of an official degree. For
example a branch certificate, a vocational education standard, a higher or an
academic education qualification standard. Certificates issued by EVC providers are
not part of higher education qualifications, but can be equivalent to higher education
qualifications.
There are two types of assessments: one for initial applicants and one for re-
assessments. Over the past years NQA carried out a few EVC-assessments:
# EVC-assessments EVC-providers (re-assessments /intital)
2017 4 - Zuyd UAS (re-assessment)
- Havenekes (initial)
- Dreesman (re-assessment)
- Trigon (re-assessment)
2016 3 - Police Academy (re-assessment)
- Leeuwenborgh (re-assessment)
- Leeuwendaal (re-assessment)
2015 5 - Zuyd UAS (re-assessment)
- Dreesman (re-assessment)
- EVC Adviesbureau (re-assessment)
- Trigon (re-assessment)
- EVC4 Gastouders (re-assessment)
2014 6 - Zuyd UAS (re-assessment)
- Christelijke Hogeschool Ede (re-assessment)
- Dreesman (re-assessment)
- EVC Adviesbureau (re-assessment)
- Trigon
2013 5 - Avans UAS (re-assessment)
- ROC Rijn IJssel (re-assessment, 2x)
- Dreesman (initial)
- EVC Adviesbureau (initial)
Total 23
7 Weblink to EVC Code 2.0
18
4.4 International degree programme assessments
The past five years our international assessments have been limited to the
assessments of degree programmes in Curaçao. The (formal) assessments all took
place in 2017. In four assessments NQA assessed several degree programmes of the
University of Curacao (UoC), formerly known as the University of the Netherlands
Antilles (UNA). Our descriptions regarding the assessments of degree programmes
throughout this self-evaluation report also apply to these international assessments as
these programmes are obliged (by their government) to comply with the Dutch
regulations for external quality assurance.
# Assessment Degree programmes
2017 4 14 degree programmes
4.5 NQA hallmark for study programmes
In addition to full degree programmes assessments, NQA also offers assessments of
components of study programmes like minors, courses and tracks. A certificate for
these study routes is an education hallmark. The hallmark can contribute to gain the
confidence of the target group in the (educational) quality of the course and supports
continual quality improvement. NQA’s assessment will be based on the national
applicable assessment frameworks, but also takes into account the norms relevant to
the programme/institution itself as well as the demands of external stakeholders. In
the past five years NQA didn’t conduct this activity.
4.6 Certification of internal audit processes
Based on the existing internal quality assurance system of an institution, the
applicable external assessment framework, NQA offers to conduct assessments on the
internal audit process of the instution. A positive outcome of this assessment coud
lead to a certificate of NQA. The assessments focusess on the quality of the audits
within an institute, gaining insight in the quality of this internal process. Over the past
five years NQA didn’t conduct a certification of internal audit processes.
4.7 Other NQA activities
Our knowledge on assessments/audits in and on higher education is valuable to us
and is respected in the sector. Attracted by this knowledge, educational institutions
and other parties contact us for consultancy services. Sometimes this actually leads to
a contract between NQA and the other party to provide a consultancy service.
Providing training is the largest component within our consultancy services.
Occasionally, NQA organises trainings/workshops for exam committees and
trainings/workshops for internal auditors. The workshops for exam committees are
related to higher education. The workshops place a special emphasis on ensuring the
quality of tests and exams. In the workshops NQA explains how, by adopting an
19
efficient and practical approach, can comply with the requirements of the Act on
Higher Education and Scientific Research.
The workshops/training of internal auditors partially regard institutions of higher
education. In the past NQA organised a couple of workshops for internal auditors of
NHL University of Applied Sciences. Currently, NQA organises workshops for auditors
of the College Zorg Opleidingen (Commission Care Education) and for internal auditors
of institutions for senior secondary vocational education and training. Next to the
trainings/workshops, NQA occasionally conducts other types of consultancy services.
This includes preparational audit/ training for the institutions that want to undergo an
institutional audit or for degree programmes, guiding institutions which want to obtain
the official status of an institution for higher education, and contributions on
conferences on specific theme, like testing and assessment.
20
5. NQA’s Quality Assurance Processes and their Methodologies
Regarding the NQA activities described in chapter 4, this chapter elaborates on the
applied processes and the methodologies put in place to assure the quality.
5.1 Assessments of degree programmes
The general outline of the process of degree programme assessments is as follows:
Phase (NQA staff) members involved
Description
Acquisition NQA account manager
The planning for the assessments of degree programmes
is dictated by the NVAO visitation roster. The NQA account
manager contacts the institutions/degree programmes to
see if degree programmes want to work with NQA as an
assessment agency. The degree programmes that want to
work with NQA as an assessment agency sign a contract.
Part of the contract is the ‘NQA Guidebook for audit visits
in higher education’, which gives an overview of the
assessment process and described the underlying steps
more in depth. Furthermore it provides guidelines for the
composition of the self-evaluation report and examples of
site-visit schedules.
Composition of the panel
NQA secretariat, assisted by a NQA project
manager
The assessment process starts with the panel
composition, based on the criteria set by the NVAO (NVAO
Panels). NQA audit panels always consist out of three
domain experts and a student member. The panels are
composed in consultation with the degree programmes.
NQA submits the panel proposal to the NVAO for approval.
When the composition is approved, the panel will be
formally installed and contracted. All panel members
receive a ‘Manual for panel members’ that in detail
describes the assessment procedure, their tasks and
obligations. This manual also provides the panel members
with instructions (training).
Coordination of the project
manager with the degree programme
NQA project managers
(auditors) & institutions/
degree programmes
The NQA account manager handed over the assessment
project to a project manager (secretary/auditor), certified
by the NVAO. This project manager is now the leading
contact person of NQA. A meeting of the project manager
with representatives of the degree programme is held to
ensure the procedures (for example regarding the self-
evaluation) are clear. In this meeting a few topics are
always addressed: the progress regarding the panel
composition, the schedule for the site-visit, required
documentation and the time-line.
Self-evaluation report
Institutions/ degree
programmes
The institution/degree programme draws up a self-
evaluation report in which it reflects on their achievements
regarding the applicable assessment framework of the
21
Phase (NQA staff) members involved
Description
NVAO (limited/extensive). The NQA Guidebook includes
suggestions for the programme for the composition of the
self-evaluation report. Six weeks before the site-visit the
degree programme forwards the self-evaluation (including
the appendices) to NQA.
Validation NQA project manager
The NQA project manager (within two weeks) screens the
content of the self-evaluation report (including
appendices) and determines if it contains the required
information. In a validation letter the project manager
approves the submitted documents or if necessary
requests extra information.
The NQA project manager forwards the self-evaluation
report and the appendices to the panel members.
Preliminary meeting of the
panel
NQA project manager & panel
members
About two weeks before the site-visit a preliminary meeting
of the panel is held. Next to instructing/training the panel
members for their task, this meeting is meant for a (first)
exchange of impressions of degree programme between
the panel members, based on the studied documentation
(self-evaluation and appendices).
Site-visit NQA project manager, panel
members & degree
programme
A site-visit is always part of the assessment process. The
schedule of the site-visit includes at least: students,
alumni, teachers/assessors, the examination committee
and the management of the programme. NQA offers
several options for the schedule for the site-visit, but the
institution is in charge of developing a suitable schedule
(checked by NQA). A specific feature of the NQA
schedules is to start the site visit with a presentation by the
degree programme. The panel uses the interviews with the
above mentioned stakeholders to verify the information
provided in the self-evaluation and gather additional
information. In addition, the panel also studies relevant
documentation, for instance learning materials, examples
of exams and assessments and minutes of Examination
Committee. The site-visit always concludes with a
notification of the provisional results of the assessment by
the chair of the panel.
Report NQA project manager & panel
members
The project manager (secretary/auditor) writes a concept
report on the basis of the submitted documents, the
outcome of the preliminary meeting and the results of the
site-visit. The findings, considerations and conclusions of
the panel regarding each standard of the prevailing
assessment framework are laid down in the report. The full
panel is asked to give feedback on the report in writing on
all standards. Every draft version of a report is also co-
read by a colleague, and if found necessary, by a third
member (in general by the NQA director).
Based on this feedback, the project manager finalizes the
draft version and this will be passed on to the degree
22
Phase (NQA staff) members involved
Description
programme, which is given the opportunity to correct
factual inaccuracies. The degree programme’s reaction is
presented to the complete panel after which the final report
will be established.
Follow-up Degree programme/ Institution/
NVAO/NQA
The degree programme/institutions submits a request for
accreditation at the NVAO based on the final assessment
report.
If a panel assesses a degree programme as ‘insufficient’
the institution can apply for an improvement period. Based
on an improvement plan the NVAO can decide to extent
the period of accreditation with two years. Within this
period the degree programme has to undergo a second
assessment in order to ensure whether it has been able to
realize the needed improvement(s).
The follow-up phase also includes the appeal procedure of
NQA (see chapter 8.7)
5.2 Research evaluations
The NQA method of working (process and methodologies) for the research evaluations
shows high similarities with the method of working for the assessments of the degree
programmes:
- a panel of independent experts (peers) is composed;
- the research unit draws up a self-evaluation report;
- the expert panel visits the research unit (site-visit);
- a report is drawn up by the independent expert panel.
NQA secretaries/project managers support the expert panels in these assessment
procedures.
As the subject of the evaluations differs from the assessments of degree programmes,
other requirements and frameworks are in place. Instead of the NVAO Accreditation
framework, the research units are assessed based upon a Branch-protocol Quality
Assurance Research (2015)8. The protocol describes in detail the procedures, the
desired requirements of the members of the expert panel, the standards for the
assessment including the criteria. NQA adopted these requirements in a Guidebook
Evaluation Research units, which includes information (incl. requirements) regarding
the process of panel composition, the self-evaluation by the research unit, the site-
visit and the process concerning the report of the panel.
8 Brancheprotocol Kwaliteitszorg Onderzoek (BKO), 2016-2022 (only in Dutch)
23
Phase (NQA staff) members involved
Description
Acquisition NQA account manager
The research unit that want to work with NQA sign a
contract. Part of the contract is the ‘NQA Guidebook
Evaluation Research units’, which gives an overview of the
assessment process and described the underlying steps
more in depth. Furthermore it provides guidelines for the
composition of the self-evaluation report and an example
for the site-visit schedule. This guidebook, together with
the Branch-protocol Quality Assurance Research form the
starting point of the research evaluations.
Composition of the panel
NQA Secretariat, assisted by a NQA project
manager
After signing the contract, the process continues with the
panel composition (commission) in accordance to the
requirements set in the branch-protocol. The commissions
are composed in consultation with the research units, to
ensure their expertise suits its profile. The commissions
consist out of three independent (research/domain)
experts. An NQA project manager supports the
commission. The experts as well as the project manager
sign a declaration of independence, which is directly
derived from the branch-protocol (annex 7 of the protocol).
Coordination of the project
manager with the degree programme
NQA project managers
(auditors) & research unit
The NQA account manager handed over the evaluation
project to a project manager (secretary/auditor). The
project manager is now the leading contact person of
NQA. A meeting of the project manager with
representatives of the research unit is held to ensure the
procedures (for example regarding the self-evaluation) are
clear. In this meeting a few topics are always addressed:
the progress regarding the commission composition, the
schedule for the site-visit, required documentation and the
time-line.
Self-evaluation report
Research unit The research unit draws up a self-evaluation report in
which it reflects on their achievements regarding the
applicable evaluation framework: branch-protocol . The
NQA Guidebook Evaluation Research units includes
suggestions for the composition of the self-evaluation
report. Four weeks before the site-visit the research unit
forwards the self-evaluation (including the appendices) to
NQA.
Validation NQA project manager
The NQA project manager (within two weeks) screens the
content of the self-evaluation report (including
appendices) and determines if it contains the required
information. In a validation letter the project manager
approves the submitted documents or if necessary
requests extra information.
The NQA project manager forwards the self-evaluation
report and the appendices to the commission.
Preliminary meeting of the
NQA project manager &
About a week before the site-visit a preliminary meeting of
the commission is held. Next to instructing/training the
24
Phase (NQA staff) members involved
Description
panel commission panel members for their task, this meeting is meant for a
(first) exchange of impressions of the research unit based
on the studied documentation (self-evaluation and
appendices).
Site-visit NQA project manager,
commission & research unit
A site-visit is always part of the process of research
evaluations. The schedule of the site-visit includes at least:
students, researchers, the management and external
stakeholders. The commission uses the interviews with the
above mentioned stakeholders to verify the information
provided in the self-evaluation and gather additional
information.
Report NQA project manager & commission
The project manager (secretary/auditor) writes a concept
report on the basis of the submitted documents, the
outcome of the preliminary meeting and the results of the
site-visit. The findings, considerations and conclusions of
the commission regarding each standard of the branch-
protocol are laid down in the report. The commission is
asked to give feedback on the report in writing on all
standards. Every draft version of a report is also co-read
by a colleague, and if found necessary, by a third member
(in general by the NQA director).
Based on this feedback, the project manager finalizes the
draft version and this will be passed on to the research
unit, which is given the opportunity to correct factual
inaccuracies. The research units’ reaction is presented to
the complete commission after which the final report will be
established.
Follow-up Research unit All research evaluations within an institution are part of the
validation process conducted by the Validation committee
quality assurance research , installed by the Association of
Universities of Applied Sciences.
Our statement that the processes and methodologies of research evaluations show a
high level of similarity with the assessments of degree programmes, makes clear that
there are also a few differences. First, the involvement of the Association of
Universities of Applied Sciences differs from the involvement of the NVAO in the
assessment process of degree programmes. For example, the association doesn’t
need to approve the commission before the start of the evaluation process. Other
differences are a logical consequence of the differences in the frameworks. But in
general, NQA’s internal processes and the applied methodologies are almost entirely
similar.
25
5.3 EVC audits
Two staff member of NQA are involved in the assessments of EVC providers. During
an EVC-audit the auditor(s) assess whether the assessment-instrument of the
provider meets the requirements of the EVC-code 2.09. Before a site-visit, the EVC-
provider has to draw up a self-evaluation report in which they describe their
compliance with the code. The assessment results are presented in a report drawn up
by the NQA auditor. The NQA Protocol EVC-assessments describes the process and
methodologies involved.
Compared to the assessments of degree programmes, the scope of the EVC
assessments is much more limited. There are two types of assessments: one for initial
applicants and one for re-assessments. The audit for initial applicants is focused on
the organization that wants to be recognized as an EVC-provider and on its work
processes. Topics within these initial audits are the professionality and integrity of the
organisation and the quality of the assessors. The initial audit can lead to registration
as EVC-provider for six months. Within this period, or as soon as ten certificates are
awarded by the EVC-provider, the organisation undergoes a re-assessment. The re-
assessment has a broader focus including aspects as the functioning, guidance and
assessment of assessors and the internal quality assurance. An important element in
the re-assessment is the check on the awarded certificates. The audit of NQA is desk-
research combined with a site-visit.
Phase (NQA staff) members involved
Description
Start of the EVC-assessment
NQA account manager / NQA
auditor
Introductory meeting NQA and applicant/EVC-provider:
leading to a contract, the establishment of a date for the
assessment and the assessment approach
Preparation of the EVC-
assessment
NQA-auditor & applicant/EVC-
provider
The applicant/EVC-provider draws up a self-evaluation and
collects the required documentation set out in the NQA
Protocol EVC-assessments. This information enables the
NQA-auditor(s) to verify compliance with the EVC-Code
2.0. The required documentation includes: an overview of
awarded certificates, a description of the EVC-assessment
procedure, résumés of the assessors, documentation on
quality assurance and their procedure for appeals and
complaints.
Site visit NQA auditor & applicant/EVC-
provider
The site-visit includes interviews with the management of
the organisation, assessors, supervisors and (if possible)
candidates.
Report NQA auditor(s) Following the format of the NKC-EVC, the NQA auditor
9 Weblink to EVC Code 2.0
26
Phase (NQA staff) members involved
Description
writes a concept report on the basis of the submitted
documents and the outcomes /results of the site-visit.
Every draft version of a report is co-read by a colleague,
and if found necessary, by a third member (in general by
the NQA director). Based on this feedback, the NQA
auditor finalizes the draft version and this will be passed on
to the applicant/EVC-provider, which is given the
opportunity to correct factual inaccuracies. The NQA-
auditor weighs the reaction of the applicant/EVC-provider
on the draft version leading to the establishment of the
finale report.
Follow-up Applicant/ EVC-provider
The applicant/EVC-provider can submit this final report to
the NKC-EVC to be or stay registered as EVC-provider.
The follow-up phase also includes the appeal procedure
of NQA.
5.4 International degree programme assessments
The international assessments of degree programme assessments concern the
assessments of degree programmes in Curaçao. The processes and methodologies
described under chapter 5.1 apply to these assessments.
5.5 NQA Hallmark for study programmes & Certification of internal audit
processes
Regarding both services of NQA, processes and methodologies are developed on the
job suiting the demands of the client. As no practice is available, NQA can not show
any evidence of applied processes and methodologies. NQA considers both to be
custom-made activities, which need to be tuned to the needs of an institution/study
programme. Consequently, the work processes, methodologies and the framework for
these activities (including the applicable quality standards) will be developed on the
job.
27
6. NQA - Internal Quality Assurance
The system of internal quality assurance is set out in NQA’s ‘Quality Management
document’ which addresses all its activities. This document reflects that the
assessments of degree programmes are the key activity of NQA.
NQA’s system of internal quality assurance is structured around the components of
the ISO 9001 and INK and addresses: ‘leadership’ (mission, vision, goals), ‘staff’,
‘resources’, ‘products, service, projects and processes’, ‘measuring and analysis’ and
‘continual improvement’. NQA organises at least two internal audits each year,
connected to these topics and our goals (targets) within. Our evaluation calendar
ensures that every topic is addressed every three years. Every year an external ISO-
certified audit takes place for the extension of our ISO-certification. In the last audit
(2016) NQA was rewarded a 4 (on a scale of 5).
As a result of our quality management our daily processes include an extensive,
valuable amount of evaluative activities (checks & balances) that safeguard the
quality and integrity of our work. The overview below highlights a few important
elements of our internal quality assurance system, including its results:
Evaluations of degree programme assessments
After every degree programme assessment, the programme receives an evaluation
form. This form generates feedback on various aspects of the assessment procedure.
It concerns (1) the cooperation with the NQA secretary/project manager, (2) the site-
visit, (3) the functioning of the expert panel and (4) the panel report. It concludes
with the question to give a grade for the performed assessment. In general the result
show that our clients are highly satisfied with our assessments. Since the introduction
of these evaluation the average score on 10 point-scale is between 7 and 8 (2014-
2017). Based on the comments in the evaluation forms we feel that the NQA approach
(focus on content of the programme) is recognized and that the cooperation with the
project managers/secretaries is a strong element. In addition, we conclude that our
work processes contribute to the satisfaction of our clients. A subject for development
regards the content of our reports. Some degree programmes feel our reports should
underpin more explicitly why their programme didn’t obtain a higher result.
Evaluations by project managers/secretaries
The project managers/secretaries of NQA fill out two types of evaluations after
conducting a degree programme assessment. First of all they fill out a project
evaluation. The outcomes of these evaluations are input for further development of
our method of working and are also input for the account managers for their
consultations with the institution. Secondly, they evaluate the members of the expert
28
panels individually. This is input for project managers that work with them in the
future, but is also valuable input for the composition of future expert panels.
Evaluations by account managers
At least once a year the account manager meets with institutions representatives to
evaluate the projects conducted that year. On a more strategic level and with the
input of above mentioned evaluations the account managers evaluates the
cooperation. The results of these evaluations could lead to a more specific approach,
tuned to the wishes of the institution, but respecting the relevant frameworks. These
consultations are also an important platform to discuss developments in the
accreditation system and the implications for future assessments.
Safeguards in the assessment procedure
To safeguard the quality and the integrity of the assessment, NQA implemented
various checks & balances in its assessment procedures. These include:
- The ‘NQA Guidebook for audit visits in higher education’ and the ‘NQA Manual
for panel members’ provide guidance for the execution of the assessments
towards the degree programmes/institutions and the panel members. These
documents support sound and intercomparable assessments.
- Checks that concern the expertise and the independence of the panel members
(further information under 4.3, ESG 3.3)
- Formats and templates as working documents for the project managers
- A meeting of the project manager/secretary with the degree programme to
ensure the procedures are clear
- A validation of the information file
- A preliminary meeting with the panel members is held one/two weeks before
the site-visit. Next to instructing/training the panel members for their task, this
meeting is meant for a (first) exchange of impressions of the degree
programme between the panel members, based on the self-evaluation report
and the appendices.
- Based on the ‘four eye-principle’ every draft report of an assessment of a
degree programme is co-read by a colleague auditor of NQA. In this process the
colleague auditor checks if the findings and considerations laid down in the
report add up to the conclusions. If necessary, a third co-reader will be included
in this process. Our procedures also ensure that the degree programmes
receive a draft version of the report. They are enabled to point out factual
inaccuracies.
29
7. ESG (Part 3) - Compliance chapter
In this chapter and in the next, NQA describes to what extent the European Standards
and guidelines apply to the system of external quality assurance in the Netherlands in
general and to NQA’s position and activities within that system in particular. The main
focus in these chapters is on the NQA activity ‘assessments of degree programmes’.
These descriptions also apply for our international assessments (Curaçao) as they are
bound by the same regulations and procedures. Concerning the NQA activities
‘research evaluations’ and ‘EVC audits’ additional information is provided in case
relevant. As NQA never conducted the services ‘certification of internal audit
processes’ and the ‘NQA Hallmark for study programmes’ it is impossible to provide
the requested information for each of ESG standards. It concerns custom-tailored
services, which in effect means that the methodologies & processes for these services
are developed in close consultation with a client (on the job). As these clients are
absent, NQA can not provide evidence required to show its compliance with the ESG
standards.
7.1 Activities, policy and processes for quality assurance (ESG 3.1)
Agencies should undertake external quality assurance activities as defined in Part 2 of
the ESG on a regular basis. They should have clear and explicit goals and objectives
that are part of their publicly available mission statement. These should translate into
the daily work of the agency. Agencies should ensure the involvement of stakeholders
in their governance and work.
All NQA activities
It is NQA’s key objective to conduct assessments of degree programmes of
universities of applied sciences in the Netherlands. It is clear that these assessments
are within the European Standards and Guidelines scope. In the past six years (2011-
2017) NQA performed almost 430 assessments to evaluate a total of 561 degree
programmes (associate degree, bachelor and master). Next to the assessments of
degree programmes, NQA offers other services (see chapter 4 and 5), but compared
to the amount of assessed degree programmes this is far less substantial: 13 research
evaluations and 23 EVC-audits.
NQA wants to emphasize that it strictly separates its external quality assurance
activities (like assessment of degree programmes) from more consultancy-like
services. All members of assessment panels and the NQA project manager (who is
formally not a panel member) involved, sign a declaration of independence which
states that in the past five years there were no (family) connections or ties of a
30
personal nature with the institution/degree programme subject in the assessment,
that could affect a fully independent judgement regarding the quality of the
programme in either a positive or a negative sense.
It is the mission of NQA is ‘to contribute to the continual improvement of the quality
of education regarding secondary education, senior secondary vocational education
and training (mbo), higher education (professional and research) and company
training’. Based on the number and quality of the performed assessments of degree
programmes in higher education and the other activities, NQA feels it works within the
scope of its mission. Unfortunately, efforts to conduct more assessments for research
universities and obtaining a stronger position in other (educational) sectors, have not
led to any effect. Therefore, our main focus stays on the professional universities. Just
recently, research universities are more open for other assessment agencies than the
one agency that had a monopoly position. As we value our contribution to the
continuous improvement of the quality of the higher professional education, we keep
focusing our attention on the quality of our (external) contributions while still being
open minded towards developing activities in other (educational) sectors and trying to
interest research universities in our assessments.
Involvement of stakeholders in governance and work
The involvement of stakeholders in our governance and work is organized in several
ways. The NVAO, the Ministry of Education, Culture & Science as well as the
Inspectorate of Education, are involved in a rule-setting manner. Their legislation on
accreditation and the underlying established assessment framework form the starting
point for our role in the system of external quality assurance regarding the quality of
the degree programmes. On a regular basis NQA meets with the NVAO to discuss the
external quality assurance system and the (practical) translation to our daily work
(processes). The past two years these discussions focused on the (implications of the)
new assessment framework and the regulatory changes that were made regarding the
panel composition, including the process of approval by the NVAO. The involvement of
the ministry and the inspectorate is, as stated, intensive on the level of the
accreditation system. The ties between NQA and the ministry and the inspectorate
are less intensive and are more thematic.
The universities of applied sciences are our main stakeholders. The NQA director, who
also manages most accounts, intensively discusses the (developments regarding) the
assessment framework with them. Based on these sessions and with the important
input of the NQA auditors, NQA yearly drafts a ‘Guidebook Audit visits in Higher
Education’. This guidebook gives an overview of the assessment process and
describes the underlying steps more in depth (see annex 2). Furthermore, it provides
guidelines for the composition of the self-evaluation report and examples of site-visit
31
schedules. As the 2016 framework is based on the notion of trust and emphasizes the
ownership of institutions and degree programmes, NQA chooses not to impose a
definite assessment approach. NQA takes on a more inviting position. In practice this
means that the NQA account managers and (later on) the NQA project managers
(secretaries/auditors) discuss a fit for purpose approach with and for each (type of) of
degree programme, taking into account the boundaries of the assessment framework.
Evaluation
NQA has formulated its mission and works accordingly, although it wishes to align its
activities a little more with the broader scope (other educational sectors) included in
its mission. Given its position as an assessment agency within the system of external
quality assurance, NQA is bound by the formal frameworks that serve as a starting
point for degree programmes assessments, but also for other activities (for example
EVC and the research evaluations). Subsequently, NQA has aligned its processes
tightly with these frameworks. The stakeholder involvement in governance and work
in general is suitable; the involvement of our clients is solid. With respect to relevant
authorities, NQA feels it (and other assessment agencies) could and should be
included more in discussions that address the (further) development of the Dutch
accreditation system. The experience of the assessment agencies with their portfolio
of performed assessments of degree programmes is not fully exploited.
7.2 Official status (ESG 3.2)
Agencies should have an established legal basis and should be formally recognized as
quality assurance agencies by competent public authorities.
Assessment of degree programmes
Since 2011, the formal position of assessment agencies (like NQA) has been
eliminated from legislation (Act on Higher Education and Scientific Research) and the
assessment frameworks set by the NVAO. A formal official status in that sense is now
absent and not strictly necessary as the NVAO is the formal decision-taking body with
regulatory effect. Contracted by institutions, assessment agencies carry out the
assessment that form the basis for that decision.
Nevertheless, in practice the position of assessment agencies in the system of
external quality assurance hasn’t changed. As before 2011, almost all assessments of
degree programmes are carried out by one of the assessment agencies mentioned on
the website of the NVAO10. The NVAO takes the NQA reports as crucial input for their
10 https://www.nvao.com/quality-assurance-systemsthe-netherlands/quality-assessment-agencies-netherlands
32
accreditation decision. Our reports are published on the NVAO website, together with
their formal decision. The summaries of our reports are adopted as body (text) for the
NVAO decision.
NQA was surprised by the decision in 2011 to remove the assessment agencies from
legislation. In time the term ‘assessment agencies’ also faded out of the assessment
framework. This removal can be seen as an attempt by the authorities to stimulate
the ownership of institutions over the assessments. Mentioning agencies in their
policies could imply that agencies have a preferred position in the accreditation
system. While in fact, institutions can organise their own external assessments as
long as they meet the NVAO criteria.
To guide the assessments NQA works with a team of internal and external
secretaries/auditors. They function as project managers, who safeguard that panels
assess degree programmes according the prevailing assessment framework. All NQA
secretaries (both internal and external) followed and completed a two-day NVAO
training to be certified.
Research evaluation & EVC audit
A formal legal status regarding these two activities of NQA is absent. Our appointment
by the National Knowledge Centre EVC is comparable to our position as an
assessment agency.
Evaluation
Although the formal status of assessment agencies changed, their position in the
system of external quality assurance has not changed. From the perspective of the
institutions/degree programmes, the authority of the assessment agencies, such as
NQA, wasn’t affected. NQA does feel that by eliminating the formal status, the
opportunities for the NVAO and other relevant authorities to verify the quality of the
assessment agencies was limited. NQA considers this a risk in the system of external
quality assurance and feels some sort of certification of the assessment agencies could
contribute to sustain the enforcement of a level playing field. NQA always advocated
this in its consultations with the NVAO. In effect, the acceptance of the outcomes of
our activities could be strengthened, although we haven’t identified fluctuations or
severe problems regarding the acceptance of our reports. At last, it is worth
mentioning that the NVAO works on criteria and processes to re-install a type of
certificating for assessment agencies.
33
7.3 Independence (ESG 3.3)
Agencies should be independent and act autonomously. They should have full
responsibility for their operations and the outcomes of those operations without third
part influence.
All NQA activities
The independency of NQA is our key asset as an assessment agency. NQA has no
organisational, or operational ties with any of the institutions it conducts assignments
for. Our procedures and the outcomes of our assessments (assessments of degree
programmes, research evaluations and EVC audits) are solely based on expertise.
NQA ensures that all assessments are carried out by independent panel members,
supported by independent NQA project managers (secretaries/auditors). All panel
members and the NQA project managers sign a declaration of independence prior to
the assessment. By signing this, they thereby declare that over the past five years
they didn’t maintain any (family) connections or ties of a personal nature or as a
researcher/teacher, professional or consultant with the institution, which could affect
a fully independent judgement regarding the quality of the programme in either a
positive or a negative sense. This is compliant with the NVAO regulations. The
declaration for the commission members of the research evaluations is slightly
different. NQA adopted the declaration that is provided within the Branch-protocol.
Furthermore, none of the panel members or involved secretaries are included in the
acquisition process leading up to the contract/agreement of NQA with the institution
that offers the degree programme. In the process of composing the panels, NQA
checks possible conflicts of interest by screening the resumes of potential panel
members. The independency of the panel (and the secretary) is also a focus point of
the NVAO, which formally approves the installation of the panel. In addition, the
independent position of the panel is strengthened by the avoidance of financial ties
between the institution and the panel. All payments (fees and expenses) are regulated
by NQA. This prevents that members of the panel feel commitment towards the
institution. This also applies to the research evaluations and EVC-audits NQA carries
out. Furthermore, during the assessments the NQA auditors are also keen on
detecting possible conflicts of interest which, in the worst case, could lead to
withdrawal of a panel/member. Fortunately, our safeguards prior to the assessment
have prevented such rigorous measures.
As NQA also provides consultancy services to institutions for higher education, a strict
separation from the external quality assurance activities is needed. Above mentioned
measures, safeguard the independency. In addition, NQA would like to point out that
none of our staff members can be involved in a consultancy project for a degree
34
programme and function within an assessment project for the same programme.
These activities are strictly separated by division between persons.
Evaluation
As independency is a key asset to NQA, NQA has put procedures in place that
safeguard this accordingly. The expertise of independent panel members is the sole
base for the realisation of our assessments of degree programmes, our research
evaluations and EVC audits. All staff and panel members are screened beforehand and
sign a declaration of independence.
7.4 Thematic analysis (ESG 3.4)
Agencies should regularly publish reports that describe and analyse the general
findings of their external quality assurance activities.
Assessment of degree programmes
In the Dutch accreditation system the NVAO is the designated body to conduct
thematic analyses. As multiple assessment agencies can be active within a visitation
group, the NVAO decides on the accreditation of all these degree programmes, based
on the peer-reviews reports of the assessment agencies. This means that the NVAO
has the overview of all conducted assessments and is in the ideal position to compose
thematic analyses.
Periodically the NVAO produces a half-yearly bulletin with an oversight of the number
of applications processed, the scores reached in assessments and short comments on
developments and trends in the last year. The NVAO’s Annual reports contain full
overviews of all decisions taken based on the peer-review reports of the assessment
agencies, such as NQA. In addition, the NVAO published a couple of thematic analyses
based on larger scale cluster-based accreditation projects. With regard to the
universities of applied sciences the NVAO published thematic analyses on the
programmes in Communication Studies (2014) and on the Teacher-training
programmes of various levels (2015 & 2016). NQA contributed to these thematic
analyses by executing the assessments of several of the degree programmes.
Moreover, for the thematic analysis on the teacher-training programmes NQA
implemented so-called ‘focus-points’ to the assessment process, which allowed the
NVAO to report on specific items in a more intercomparable way.
As NQA is commissioned by the institutions/degree programmes, their demand
regarding the assessment process is leading (within the boundaries of the NVAO
Framework). As for most institutions and degree programmes the assessment already
35
is an intensive process, the demand for additional services (such as thematic analysis)
is very limited, partly because these additional services are not financed. In 2012 NQA
carried out a thematic analyses for eight bachelor programmes of nursing, in 2015-
2014 for masters Advance Nursing Practice and in 2013 for three bachelor
programmes Midwifery.
For multiple universities of applied science, NQA organizes (yearly) lectures on the
most recent developments and to share our experiences regarding the assessment
procedures. And, twice a month (on average) NQA is invited to meetings of visitation
groups, to inform the participating degree programmes on their upcoming
accreditation process.
Furthermore, NQA participates active in consultations of various bodies within the
system of higher education to share its knowledge and impressions acquired in the
assessments. For example, NQA participated in two thematic research projects of the
Inspectorate of Education regarding examination committees and education
committees. NQA also shares its experiences in thematic congresses/workshops, such
as congresses on topics like testing. And occasionally, publishes on a well-known
website about higher education (www.scienceguide.nl), for example on the newest
assessment frameworks11.
Research evaluation & EVC audit
Regarding the research evaluations and the EVC audits NQA feels it is not in the
position to conduct thematic analysis as meant in this standard (published reports).
Besides, it feels that the amount of assessments is too limited to present general
findings. NQA does share its knowledge obtained in assessments in various
consultations, for example with the National Knowledge Centre EVC.
Evaluation
Regarding the degree programme assessments, the NVAO is the designated body to
conduct thematic analysis. NQA feels it has a modest role as an unfunded body,
regarding this standard of the ESG. The system of accreditation offers the opportunity
to carry out thematic analysis and in practice various analysis have been executed;
mostly by the NVAO but with input of the NQA peer-review reports. As, in NQA’s
opinion, these thematic analysis stimulate the learning effect of all carried out degree
programme assessments, NQA would welcome further use of this instrument by the
NVAO. With regard to the research evaluations and EVC, NQA feels the amount of
assessments is too limited to present general findings. A practical issue concerns the
ownership of the assessment reports of our activities. The institutions are the formal
11
http://www.scienceguide.nl/201603/vraagtekens-bij-borging.aspx
36
owner of the assessment reports. Before the reports become public, NQA can’t use the
information for thematic analysis unless it has permission of the institution. It can
take up to 1,5 year before the reports are publicly available.
7.5 Resources (ESG 3.5)
Agencies should have adequate and appropriate resources, both human and financial,
to carry out their work.
All NQA activities
NQA is an independent, private organisation and thus doesn’t receive any form of
funding form the Dutch government or other governmental institutions or
organisations. Our financial resources depend on the fees we manage to receive for
the assessments of degree programmes and our other activities. And so, our revenues
are strictly connected to the amount of assessments/assignments we acquire. As our
solid key activity is the assessment of degree programmes, our yearly financial figures
are highly influenced by the visitation schedule, which is regulated by the NVAO.
The composition of our workforce has been relatively stable in its size over the past 3
years. It consists of a secretariat of three committed team members, a team of seven
internal and a flexible group of seven project managers/auditors (secretaries of the
assessment panels) and a director. Our project calendar is leading for decisions on
recruitment of new staff members. Every new (internal) staff member follows the NQA
introductory program. Furthermore, our quality assurance policy ensures all staff
members have a performance evaluation once a year. These evaluations are meant to
reflect on the activities of the staff member of the past year and look forward to the
year to come. Further the performance evaluations are focussed on professional
development. In the end all this leads to specific activities related to professional
development, in accordance to the ambitions of NQA. For example: the last one to two
years a staff member attended a series of conventions/conferences on applied
research before carrying out research evaluations. Several project managers
participated in conferences on testing in higher education. And one project manager is
supported to obtain is master degree Public administration and Organisation Science
(Utrecht University).
One of our internal project managers co-ordinates the external project managers. In
addition, our secretariat is available to them for support.
37
Evaluation
NQA has a solid financial basis and a committed and competent team, which is
adequate and appropriate for carrying out its activities. The visitation schedule highly
influences our project calendar. This schedule is not ideal in terms of equal spread,
but NQA manages to maintain its consistent qualitative profile. A larger external group
of NQA auditors was set up, to deal with peaks in the project calendar over the years.
Nevertheless, NQA feels that a more equal spread of the degree assessments would
have been possible and could have contributed to the quality of assessments. With a
broader portfolio, with services like research evaluations and EVC, NQA tries to be less
vulnerable for the valleys in the visitation schedule.
7.6 Internal quality assurance and professional conduct (ESG 3.6)
Agencies should have in place processes for internal quality assurance related to
defining, assuring and enhancing the quality and integrity of their activities.
All NQA activities
NQA has an appropriate system of internal quality assurance in place defining,
assuring and enhancing the quality and integrity of its activities. This system of
internal quality assurance is set out in NQA’s Quality Management document which
addresses all its activities. This document reflects that the assessments of degree
programmes are the key activity of NQA. NQA’s system of internal quality assurance is
structured around the components of the ISO 9001 and INK and addresses:
‘leadership’ (mission, vision, goals), ‘staff’, ‘resources’, ‘products, service, projects
and processes’, ‘measuring and analysis’ and ‘continual improvement’. NQA organizes
at least two internal audits each year, connected to these topics and our goals
(targets) within and following our evaluation calendar ensuring that every topic is
addressed in a period of three years. Every year an external ISO-certified audit takes
place for our ISO-certification.
Next tot these evaluations NQA embodied an extensive amount of other (evaluative)
activities (checks & balances) to safeguard the quality and integrity of its activities.
They include:
- The ‘NQA Guidebook for audit visits in higher education’ and the ‘NQA Manual for
panel members’ provide guidance for the execution of the assessments towards the
degree programmes/institutions and the panel members. These documents support
sound and intercomparible assessments.
- In bi-weekly team meetings and in occasional topic meetings NQA discusses the
developments in the sector, in the system of accreditation, the functioning of the
38
audit panels, the developments regarding other NQA activities (for example
research evaluations and EVC) and other important issues (such as the introduction
of the 2016 NVAO Framework or procedures regarding the composition of the audit
panels)
- Consultations with external parties, like the NVAO, the NKC-EVC and institutions for
higher education
- Evaluations by the account managers. A meeting once a year with the institutions
for higher education (clients), in which all NQA activities (assessment of degree
programmes and research evaluations) are subject of evaluation on a strategic
level.
- Based on the ‘four eye-principle’ every draft report of an assessment is co-read by a
colleague auditor of NQA. In this process the colleague auditor checks if the findings
and considerations laid down in the report add up to the conclusions. If necessary, a
third team member will be included in this process. Our procedures also ensure that
degree programmes receive a draft version of the report. They are enabled to point
out factual inaccuracies (see also: ESG 2.5). This procedure is similar for the
research evaluations and EVC-audits.
- After finalizing the assessment procedure every degree programme receives an
evaluation form in which it can give feedback on the aspects of the assessment
project.
- Auditors of NQA fill out two types of evaluations. First of all they fill out a project
evaluation. The outcome of these evaluations are input for further development of
our method of working but are also input for the before mentioned evaluations by
account managers. Second, they evaluate the members of the panels individually.
Evaluation
NQA feels its procedures for internal quality assurance are highly appropriate. As also
becomes clear in our ISO-results. It leads to improvements in our method of working,
but also contributes to the fine-tuning of our procedures, processes and underlying
documents. Over the past few years the administrative burden of accreditation has
been a recurring topic. The installment of the institutional audit within the
accreditation system was supposed to have a positive effect (more effective and less
time-consuming) on the perceived administrative burden. In practice the effect on the
perceived administrative burden was minimal. Especially because the institutions felt
an extra audit was added to the system. Although NQA has some reservations
regarding this topic, it worked actively to find ways to cut back the experienced
administrative burden, while safeguarding the quality of the assessments.
39
7.7 Cyclical external review of agencies (ESG 3.7)
Agencies should undergo an external review at least once every five years in order to
demonstrate their compliance with the ESG.
All NQA activities
This is NQA’s initial application for ENQA membership, meaning that there is no
previous audit. As the guidelines state “a periodic external review will help the agency
to reflect on its policies and activities”. Appreciating input form external reviewers, it’s
NQA’s ambition to undergo an external review at least once every five years in order
to demonstrate our compliance with the ESG.
40
8. ESG Part 2 - Compliance chapter
8.1 Consideration of internal quality assurance (ESG 2.1)
External quality assurance should address the effectiveness of the internal quality
assurance processes described in Part 1 of the ESG.
Assessment of degree programmes
The Dutch system of accreditation reflects the institutions’ responsibility for the
quality of their programmes. The system in itself reflects this consisting of the
institutional audit combined with limited or extensive assessments (if there is no or
not a positive result of an institutional audit) of the degree programmes. It assumes
that institutions in general and degree programmes in particular organise effective
periodic feedback that supports the achievement of the intended learning outcomes.
The outcomes of evaluations/feedback should demonstrably constitute the basis for
development and improvement.
The elements of ESG Part 1 are more explicit embedded within the 2016 NVAO
Assessment Framework, as stated by the NVAO. This becomes clear in the table
below12, which shows which standards within the assessment frameworks (2016)
cover the respective aspects and elements of part 1 of the ESG:
ESG-standards,
part 1
2016 Assessment
Framework
Institutional audit
2016 Limited
Assessment
Framework degree
programmes
2016 Extensive
Assessment
Framework degree
programmes
1. Policy and
procedures for
quality assurance
1 and 2 * (means, covered
within institutional
audit)
9
2. Design and
approval of
programmes
2 1 and 2 1, 2, 3 and 9
3. Student-centred
learning, teaching
and assessment
1 2, 3 and 4 4, 10 and 11
4. Student
admission,
progression,
recognition and
certification
2 2 5, 8 and 11
12 Derived from the NVAO Self-evaluation report for their ENQA-review (adjusted)
41
5. Teaching staff 2 2 6
6. Learning
resources and
student support
2 2 7 and 8
7. Information
management
3 and 4 * 7
8. Public
information
2 2 8
9. Ongoing
monitoring and
periodic review of
programmes
3 and 4 * 9
10. Cyclical
external quality
assurance
3 and 4 * 9
Research evaluation & EVC audit
There is a crucial difference between the assessments of degree programmes and the
research evaluations and EVC audits. The degree programmes object in the
assessments of degree programmes, lead to a formal degree. This is not the case with
the research evaluations and the EVC-audits. Although both (research units & EVC-
providers) are part of the system of higher education, they don’t offer a (degree)
programme for students with intended learning outcomes, a teaching and learning
environment and tests and assessments to see if students obtained the intended
learning outcomes. In effect, the focus within these two types of assessments is
different and not all aspects addressed in ESG 1 are included in the assessments. The
following overview shows the respective aspects embedded (translated to its context):
Research evaluations EVC audits
ESG-standards, part 1 Branch protocol 2015 EVC-code 2.0
1. Policy and
procedures for quality
assurance
5 5
2. Design and approval
of programmes
1, 3 1, 3
3. Student-centred
learning, teaching and
assessment
- -
42
Research evaluations EVC audits
4. Student admission,
progression,
recognition and
certification
- -
5. Teaching staff 2 4
6. Learning resources
and student support
- -
7. Information
management
- 2
8. Public information 4 2
9. Ongoing monitoring
and periodic review of
programmes
5 5
10. Cyclical external
quality assurance
5 5
Evaluation
Part 1 of the ESG is well integrated in the Dutch system of accreditation, i.e. in the
frameworks for the assessments on an institutional level and on the level of the
degree programmes. Compliance with part 1 of the ESG is primarily the responsibility
of the institutions and its degree programmes. As an assessment agency within the
system of external quality assurance, NQA assesses the way these aspects are
embedded in the degree programmes. On this ‘system level’ it is clear how the
standards of part 1 (ESG) are embedded in the Dutch Assessment Framework. In
daily practice, panel members as well as institutions/degree programmes are limited
aware of the ties between the two. This awareness could be strengthened.
The compliance of the frameworks for the research evaluations and the EVC audits
with ESG part 1 is less strong, although most aspects are covered. The connection of
the underlying frameworks with the student-related standards of Part 1 is the weakest
element, mainly because the object of the assessments (research units and EVC-
providers) aren’t focused on educating students. Nevertheless, a strong emphasis on
(internal) quality assurance is embedded.
43
8.2 Designing methodologies fit for purpose (ESG 2.2)
External quality assurance should be defined and designed specifically to ensure its
fitness to achieve the aims and objectives set for it, while taking into account relevant
regulations. Stakeholders should be involved in its design and continuous
improvement.
Assessment of degree programmes
The 2016 Assessment Framework for degree programmes is developed by the NVAO
in close consultation with, among others, the Ministry of Education, Culture and
Science. This most recent revision of the assessment framework is set in motion in
2014, with the installation of an advisory group. This advisory group consisted of
representatives of the umbrella organizations of universities (both professional and
research), the NVAO, student organizations, the Inspectorate of Education, the
employers and the trade unions. It were the conclusions and recommendations of this
advisory board that led to an assessment framework (2016) based on a notion of trust
and a stronger emphasis on ownership of institutions and degree programmes.
The structure of the accreditation system and the assessment the framework doesn’t
differ much from the previous version(s). The system introduced in 2011 with an
institutional audit and separate assessments of degree programmes is still the basic
structure of the Dutch system of external quality assurance. Institutions that hold a
positive or conditionally positive decision regarding its institutional audit can use the
limited framework for their assessments of degree programmes. In all other cases,
the extensive framework is dictated. With the introduction of the institutional audit it
was assumed that the workload involved in a degree programme assessment would
be reduced as aspects were already assessed in the institutional audit. With the
introduction of the 2014 framework further reduction was intended particularly by
shortening the list of compulsory documents for the assessment procedures. These
attempts show that creating fitness for purpose is a continual process. In
consultations with the NVAO and other authorities, NQA expresses its opinion on the
development of the system of accreditation and particularly on the revisions of the
assessment framework.
On a project level the NQA project managers ensures a fit for purpose assessment
approach. With the ‘NQA Guidebook’ (based on the prevailing framework) as a solid
starting point. In a coordination meeting with representatives of the degree
programmes the project manager discusses for example the schedule for the site-visit
and he required documentation to tune the design of the assessment to the
characteristics of the degree programme (see also: chapter 5.1).
44
Research evaluation & EVC audit
Regarding the research evaluations a similar system as used for the assessments of
degree programmes is in place. The Branch protocol is leading in the application of
methodologies; NQA follows these requirements. Concerning the EVC-audits the EVC-
Code 2.0 is the starting point. With the NKC-EVC NQA discussed its assessment
approach as well as with its clients. Establishing the type of assessment an EVC-
provider needs to undergo, is leading in the applied methodologies.
Evaluation
NQA consequently adopts the continuous development of the assessment framework
of the NVAO and translates the changes into its work procedures (for example: yearly
update of our Guidebook). It is clear that the external quality assurance for degree
programmes takes the relevant regulations (as laid down in the Act on Higher
Education and Scientific Research) as its starting point. Attention goes out to ensure
that the methodologies applied are fit for purpose. It has to be said that NQA feels
this is a continuous and ongoing process. Today, the former strong confidence in the
institutional audit seems to be on a turning point. Several universities for applied
sciences feel that the benefits of a limited assessment of their degree programmes
don’t weigh up to the workload of the institutional audit. Others embrace the notion of
trust and emphasis on ownership in the 2016 framework, but as a result feel that
their degree programmes should themselves (as owners) show their quality in
external quality assurance procedures. NQA follows and participates in these
discussions, to contribute to an even more fit for purpose system of external quality
assurance. An example of our participation in these discussions is the article13 NQA
published on the (very doubtful) connection between the outcomes of the institutional
audits and the outcomes of the assessments of degree programmes. On a project
level the coordination meeting with representatives of the degree programmes
ensures a fit for purpose assessment approach.
Regarding the research evaluations and EVC audits NQA works within the boundaries
set in the relevant frameworks and tunes its methodologies based on consultations
with the research units/EVC-providers.
13
http://www.scienceguide.nl/201411/toets,-kwaliteit-en-borging.aspx
45
8.3 Implementing processes (ESG 2.3)
External quality assurance processes should be reliable, useful, pre-defined,
implemented consistently and published. They include:
- a self-assessment or equivalent;
- an external assessment normally including a site visit;
- a report resulting from the external assessment;
- a consistent follow-up.
Assessment of degree programmes
As set in the 2016 Assessment Framework, as well as in previous versions of the
framework, a self-evaluation, a site-visit of the audit panel and an assessment report
are part of the process of external quality assurance. These aspects are solidly
translated into NQA’s Guidebook Audit visits in Higher Education. The degree
programme/the institution sends the assessment report to the NVAO, that takes a
decision regarding the (re)accreditation of the programme, based on the conclusions
of the audit panel.
Self-assessment
Based on the NVAO’s assessment framework and NQA’s Guidebook, the institution
draws up a self-evaluation report describing the programmes strengths and
weaknesses. The report should be a self-contained document (free format) comprising
a maximum length of fifteen pages (limited framework) or twenty pages (extensive
framework), addressing the standards of the prevailing framework. The programme
appends a limited number of appendices to its self-evaluation. These appendices
provide insight into the set-up and content of the curriculum, the composition of the
staff team, and the teaching and examination regulations. In addition the panel
selects fifteen graduates. The products of these graduates, that in the vision of the
programme showcases the realisation of the intended learning outcomes, are also part
of the information provided to the panel. Directly after the information is available (six
weeks before site visit), NQA checks the content, before sending the information-file
to the members of the panel for their preparation. A preliminary meeting with the
panel members is held one/two weeks before the site-visit. Next to
instructing/training the panel members for their task, this meeting is meant for a
(first) exchange of impressions of the degree programme between the panel
members, based on the self-evaluation report and the appendices.
Site-visit
A site visit is always part of the assessment process. The duration of the site visit is
usually one day, unless the amount of degree programmes within an assessment
requires extension of the duration. NQA chooses to include all members of the panel
46
in the visit the degree programme. The institution is in charge of developing a suitable
(checked by NQA) schedule for the site visit, which includes at least: students,
alumni, teachers/assessors, the examination committee and the management of the
programme. NQA offers several options for the schedule of the site visit, which in
practice are used/adopted by the degree programmes to determine a suitable layout.
A specific feature of the NQA schedules is to start the site visit with a presentation by
the degree programme. In this presentation highlights of the programme as well as
strengths and weaknesses can be explained and put into context. With the
implementation of the 2016 Assessment Framework, a new item in the site visits is
the so called development dialogue. NQA implemented this in its method of working.
Report & follow up
The findings, considerations and conclusions of the panel are laid down in a report
(see also: next paragraph peer-review reports). The follow up of the assessment is
that the NVAO takes a decision about the accreditation of a degree program that is
based on the conclusion as laid down in the peer review report. If a panel assesses a
degree program as ‘insufficient’ - and if the NVAO subsequently decides so - the
institution can apply for an improvement period. Based on an improvement plan the
NVAO can decide to extent the period of accreditation with two years. Within this
period the degree programme has to undergo a second assessment in order to ensure
whether it has been able to realize the needed improvement(s).
Research evaluation & EVC audit
NQA uses a similar approach for the research evaluations and the EVC-audits as
described above (see also chapter 5). A few aspects a slightly different:
- Interviews with stakeholders: following the focus of the assessments, the panels
interview other stakeholders groups in the assessments. The site visit within the
research evaluations holds interviews with researchers, the management, external
stakeholders and students. Within the EVC audits the management of the
organisation, assessors, supervisors and (if possible) candidates are interviewed.
- The follow-up: regarding the research evaluations the report becomes part of an
audit by the Validation committee quality assurance research on an institutional
level. The report as a result of an EVC audit can be used by the EVC-provider to be
registered by the NKC-EVC.
Evaluation
The external quality assurance processes include a self-assessment, a site-visit, a
peer review report and a consistent follow-up. NQA consistently adopts the
assessment framework(s) of the NVAO, the Branch Protocol and the EVC-Code 2.0
and translates this to internal (working) processes that contribute to a reliable and
useful execution. By standardizing our internal work processes, including checks and
47
balances, instructions for the degree programmes (guidebooks, protocols), selection
and training of the panel members for assessments of degree programmes (manual
for panel members) and the training of our auditors, NQA feels confident its
assessments of degree programmes, research evaluations and EVC-audits are carried
out professionally, consistently and transparently.
In general but specifically regarding the assessments of degree programmes, NQA
wishes to handle the content of the degree programmes as the central focus of our
assessments. Based on the evaluations held under assessed degree programmes, the
feedback in account-evaluations and the feedback of the NVAO, NQA feels it succeeds
in this ambition.
A nowadays revitalized discussion addresses the assessment scales within the NVAO
frameworks (unsatisfactory, satisfactory, good and excellent). It is argued that the
accreditation isn’t the proper instrument (in terms of reliability) to differentiate
between satisfactory, good and excellent as its key objective is to assure degree
programmes meet the basic quality standards regarding content, level and
orientation. All assessment agencies advised the NVAO to adopt a two-scale system
(pass/fail). This discussion is part of a political debate that continues in coming
months.
8.4 Peer-review experts (ESG 2.4)
External quality assurance should be carried out by groups of external experts that
include (a) student member(s).
Assessment of degree programmes
The NQA audit panels always consist out of three domain experts and a student
member. For the composition of the audit panels, NQA has a work process in place
that contains the regulations set by the NVAO. The members of the panels are
independent and experts in the discipline concerned, in education, the professional
field, assessment aspects, and in student issues. In addition, the panel commands
international expertise and experience. Every panel member has to sign of a
declaration of independence, which states that the panel member doesn’t have
(family) connections or ties of a personal nature with the institution/degree
programme subject in the assessment in the past five years. The requirements ensue,
as the NVAO states, from the Standards and Guidelines for Quality Assurance in the
European Higher Education Area (European Standards and Guidelines, ESG) and are
formulated in the spirit of the requirements set at the European level for such panels
in higher education. NQA adopted these requirements in its process to compose the
48
panels. NQA submits its panel proposal for approval to the NVAO. When the
composition is approved, the panel will be formally installed and contracted. Although
NQA has a severe, guiding role in this process, it has to be made clear that the
composition of the panel is primarily the responsibility of the institutions.
All panel members receive a ‘Manual for panel members’ that in detail describes the
assessment procedure, their tasks and obligations. This manual also provides the
panel members with instructions (training) to support a valuable assessment
procedure. For example these instructions include information on the standards (of
the NVAO Assessment Framework) that are subject in the assessment, the definitions
of judgement (assessment criteria), but also address the desired attitude for the
interviews during the site visit and provides specific instructions for the chairman.
During the preliminary discussion (one/two weeks before the site visit) this training
concludes with the instructions of the involved NQA auditor. Chairs of the panels sign
a document that stating they are instructed/trained for their task.
Research evaluation & EVC-audit
For the research evaluations a similar practice is in practice concerning the
composition of the panel. It consists of three independent experts (peers) that
represent expertise on applied research, related education and the professional field.
Regarding the EVC-audits the panel consists of one or two NQA auditors. Their
knowledge and skills is approved by the NKC-EVC as it appointed NQA for assessing
EVC-providers. A student member is formally not obliged in these two types of
assessments.
Evaluation
NQA believes its practice is compliant with ESG 2.4. The quality of audit panels is our
calling card towards the institutions/degree programmes (including research units and
EVC-providers) NQA assesses. NQA takes on a severe, supportive role in the process
of the panel composition, but within the assessments of degree programmes and the
research evaluations it is primarily the institutions responsibility. In general the
institutions are satisfied with the quality of the panels. Their involvement in the
composition process is appreciated and strengthens their commitment to the
outcomes. Furthermore, our intensive evaluations show that the level of expertise in
the panels is good as are their communication techniques. Based on their expertise
and own experiences the input of certain panel members can be one-sided (pet-
subject), NQA wishes to prevent this situation and invests in this by addressing it in
the preliminary discussion of the audit panel. Also, the evaluations show us that the
input of the student member of the panel could be enriched.
49
As a result of growing requirements of the NVAO for the expertise in the audit panels
for degree programmes, as well as the growing complexity of composing the panels
due to the desired overlap in panels within a cluster of degree programmes, NQA pays
a lot of attention to this aspect of the assessment process. It has become a bigger
challenge to represent the requirements and overlap with the audit panels of degree
programmes within their cluster. By intensifying the deliberation with our colleague
assessment agencies and the extension of the timeline for the composition, NQA
manages to install panels that represent all requirements. That being said, the
process of approval by the NVAO with respect to the dates of the site visits within a
cluster of degree programmes, led to difficulties: more bureaucracy and degree
programmes feel there is less opportunity to be assessed by a righteous peer-panel
due to the overlap-regulations. To overcome this, the process of composing the panels
as well as the involvement of other partners in this process, are discussed. NQA (as
well as other assessment agencies) addressed this issue multiple times in
consultations with the NVAO.
8.5 Criteria for outcomes (ESG 2.5)
Any outcomes or judgements made as a result of external quality assurance should be
based on explicit and published criteria that are applied consistently, irrespective of
whether the process leads to a formal decision.
Assessment of degree programmes
The criteria set by the NVAO in the assessment framework are leading in the
judgements of our audit panels on the quality of the degree programmes. The criteria
are published on the NVAO website and are adopted in the ‘NQA Guidelines for audit
visits of degree programmes’ and in the Manual for panel members. The following
assessment rules apply:
Judgement per standard: Assessment rules for overall
judgement on the degree
programme (limited):
Assessment rules for overall
judgement on the degree
programme (extensive):
The panel scores each standard
according to the following scale:
unsatisfactory | satisfactory |
good | excellent
The panel comes to an overall
judgement on the programme on
the following scale:
unsatisfactory | satisfactory |
good | excellent
The panel comes to an overall
judgement on the programme on
the following scale:
unsatisfactory | satisfactory |
good | excellent
Definition scores for judgement
on standards:
- Unsatisfactory: the
programme does not meet the
generic quality standard and
Rules for overall judgement
(limited assessment):
- Excellent: a judgement of
excellent with respect to at
least two standards, one of
Rules for overall judgement
(extensive assessment):
- Excellent: a judgement of
excellent with respect to at
least five standards, one of
50
shows shortcomings with
respect to multiple aspects of
the standard.
- Satisfactory: the programme
meets the generic quality
standard across its entire
spectrum.
- Good: the programme
systematically surpasses the
generic quality standard
- Excellent: the programme
systematically well surpasses
the generic quality standard
and is regarded as an
international example.
Definition generic quality: the
quality that, in an international
perspective, may reasonably
be expected from a higher
education associate degree,
bachelor’s or master’s
programme.
which must be standard 4, and
a judgement of at least
satisfactory with respect tot
the remaining standard.
- Good: a judgement of good
with respect to at least two
standards, one of which must
be standard 4, and a
judgement of at least
satisfactory with respect to the
remaining standards.
- Satisfactory: a judgement of
at least satisfactory with
respect to at least two
standards, one of which must
be standard 1, and
improvement of the
shortcoming(s) identified
under the standards scored
unsatisfactory must be
realistic and feasible within
two years (improvement
period)
- Unsatisfactory: i) standard 1
is scored unsatisfactory; or ii)
one or two standards are
scored unsatisfactory and
improvement within two years
is neither realistic nor feasible;
or iii) three or more standards
are scored unsatisfactory.
which must be standard 11,
and a judgement of at least
satisfactory with respect tot
the remaining standard.
- Good: a judgement of good
with respect to at least five
standards, one of which must
be standard 11, and a
judgement of at least
satisfactory with respect to the
remaining standards.
- Satisfactory: a judgement of
at least satisfactory with
respect to at least six
standards, one of which must
be standard 1, and
improvement of the
shortcoming(s) identified
under the standards scored
unsatisfactory must be
realistic and feasible within
two years (improvement
period)
- Unsatisfactory: i) standard 1
is scored unsatisfactory; or ii)
at least six standards are
scored unsatisfactory and
improvement within two years
is neither realistic nor feasible;
or iii) less than six standards
are scored satisfactory.
Source: Assessment framework, NVAO (2016)
To assure a consistent interpretation and application of the criteria, NQA discusses
them with its auditors, with panel members and with colleague assessment agencies.
NQA cooperates with other assessment agencies (in the cluster groups), for example
we organized calibration sessions in which the NVAO criteria were discussed. These
discussions led to a more in depth description of the definitions of the criteria that are
adopted in our guidebook and manual for panel members. As part of our internal
quality assurance, every draft version of a report is co-read by a colleague (four-eye
principle). The use of the criteria is part of this internal check system.
NQA is not the body that decides on the (re)accreditation of degree programmes. The
NVAO takes the report of the audit panel (that includes their considerations,
judgements and recommendations) as a starting point of its own assessment,
51
verifying whether the panel has convincingly argued its judgements and whether it
has carried out the assessment in accordance with the NVAO’s guidelines. In the vast
majority of cases, the NVAO follows the outcomes of the assessments by the NQA
audit panels. In a small number of cases, the NVAO asks an audit panel for additional
information on the judgements made in their peer review report. In the last six years
all judgements were accepted by the NVAO. In just one case (2010) the NVAO
decided to conduct a verification.
Research evaluation & EVC audit
The criteria laid down in the branch protocol (research evaluations) and in the EVC-
Code 2.0 (EVC-audits) are leading in the judgements of our audit panels on the
quality of the research unit or EVC-provider. The criteria for the research evaluations
are published on the website of the Association of Universities of Applied Sciences.
The criteria for the EVC audits are published on the website of the NKC-EVC. The
criteria are adopted in our internal work processes (protocol).
Following the criteria of the branch protocol a panel assesses the first four standards
on the following scale: unsatisfactory, satisfactory, good or excellent. Standard 5 is
assessed unsatisfactory or satisfactory. The overall judgement can only be positive if
standard 3 is assesses satisfactory. Regarding the EVC audits all five codes have to be
met by the EVC-provider. Each code is subdivided in criteria: 29 in total.
Evaluation
The judgements of the NQA audit panels are clearly based on explicit and published
criteria. The prevailing NVAO framework that forms the solid starting point of the
assessment is part of the contract that an institution/degree programme signs with
NQA. Furthermore, the NVAO criteria are adopted in the ‘NQA guidelines for audit
visits of degree programmes’ and in the ‘Manual for panel members’. This includes a
further elaboration of the NVAO criteria to assure a consistent interpretation.
In a similar way NQA adopted the criteria that are laid down in prevailing frameworks
for the research evaluations and the EVC audits.
In the assessment procedure of NQA, institutions/degree programmes (including
research units and EVC-providers) are enabled to point out factual inaccuracies in a
draft version of the peer review report. Occasionally, this step in the assessment
procedure leads to discussion on the interpretation of the criteria. Especially, when
standards within the framework are found ‘unsatisfied’. Mostly, a more extensive
description or explanation of the shortcoming lead to understanding and acceptance
by the degree programmes of the judgements.
52
8.6 Reporting (ESG 2.6)
Full reports by the experts should be published, clear and accessible to the academic
community, external partners and other interested individuals. If the agency takes
any formal decision based on the reports, the decision should be published together
with the report.
Assessment of degree programmes
Taking into consideration that the NVAO is the formal decision-taking body within the
Dutch system of external quality assurance, it is their obligation to publish the peer
review reports as well as their formal decision based on that (advisory) report. As
stated before, these peer review reports are produced by assessment agencies such
as NQA and serve as the basis for the NVAO decisions on (re)accreditation. The NVAO
website provides a publicly available and searchable database that contains links to
the decision and the underlying peer review reports. NQA itself does not publish
reports on degree programmes, because of confidentiality and the fact that formally
the institution/degree programme is the owner of the report.
The NQA peer review reports include:
- a context description of the institution/degree programme: for example: orientation
and level of the programme;
- a description of the individual procedure, including the experts involved: for
example: applicable NVAO framework and the names and roles within the
assessment procedure of the panel members;
- evidence, analysis and findings: based on the applicable NVAO framework the
findings, considerations and judgements of the audit panel form the main
component of our peer review reports;
- conclusions; in a separate chapter the audit panel weighs its judgements on the
standards of the applicable framework to come to a conclusion on the overall
judgement of the degree programme (based on the assessment rules for overall
judgement on the programme set in the NVAO framework);
- if applicable features of good practices, demonstrated by the institution;
- recommendations for follow-up action: in a separate chapter the audit panel draws
up its recommendations logically derived from their findings on their judgements on
the standards.
The NQA peer review reports furthermore contain a summary and before finalizing the
peer review report, the degree programme has the opportunity to point out factual
inaccuracies in the draft version of the report.
53
Research evaluation & EVC audit
The research units (or its institution) and EVC-providers are the owner of the reports.
It is up to them to decide whether they want to use it in their process to be validated
by the Validation committee (VKO) or to be/stay registered as an EVC-provider. If the
answer to these questions is positive, the reports become part of process with another
body: VKO or NKC-EVC. Regarding the EVC-audits for example the NKC-EVC has a
website that contains all registered EVC-providers14.
Evaluation
As a result of the arrangement of the Dutch system of external quality assurance,
NQA does not publish the reports of their carried out assessments of degree
programmes. The NVAO is the formal decision-taking organisation within this system
and so the designated body to publish the decision on (re)accreditation and the
underlying peer review reports (of assessment agencies like NQA). The decision and
reports are publicly available on the website of the NVAO (www.nvao.net). In self
reflective perspective, NQA feels it can do more to unlock the publicly available
decisions and the underlying reports of assessments of degree programmes,
especially its own peer reviews reports within. NQA takes into consideration to create
a weblink (or if possible direct access-link) to the NVAO database.
8.7 Complaints and appeals (ESG 2.7)
Complaints and appeals processes should be clearly defined as part of the design of
external quality assurance processes and communicated to the institutions.
Assessment of degree programmes
NQA has formal procedures in place for complaints and appeals by institutions/degree
programmes regarding the assessment of the degree programmes. This appeal
procedure has the form of ‘hear both sides’ consultation and aims at reaching
consensus on the facts and mutual understanding about the assessment. This meeting
can lead to the following outcomes, described in the ‘NQA Guidelines for audit visits in
higher education’:
- the degree programme accepts NQA’s judgement;
- the audit panel has overlooked or misinterpreted prior received documents or those
laid out for inspection. The panel re-appraises specifically those standards that have
been negatively assessed. The panel and the programme make a record of what is
to be re-appraised, how it will be implemented and according to what time schedule.
14
https://www.ervaringscertificaat.nl/evc/aanbieders
54
- NQA will postpone the final verdict to allow the programme to make corrections.
NQA and the programme set down in writing how this will happen and according to
what time schedule.
- NQA amends the assessment.
- NQA abides by its verdict although this is unacceptable to the programme. The
program is then free to request a second opinion.
Furthermore, NQA’s Quality Management document provides the handling of
complaints. It differentiates internal complaints of staff members and external
complaints. The procedure for internal complaints is elaborated in NQA’s Personnel
Manual. External complaints are appointed to the NQA account manager of the
institution or the director of NQA. An inventory consultation is set up followed by
possible follow-up arrangements. If the complaint issues the peer review report of the
assessment procedure, the above mentioned appeal procedure comes into force.
The coming into force of the appeal procedure and the procedure for complaints is a
rarity. However NQA feels that the last two/three years institutions/degree
programmes tend to be less hesitant to seek procedures like these to influence the
outcome of the assessment. Especially, when the overall judgement of the
assessment is insufficient. In the past three years three appeal procedures were set in
motion. In one case this led to a mutual agreement and understanding; in another the
procedure was withdrawn by the institution. Unfortunately, in another case a mutual
agreement proved to be impossible and is a legal procedure inevitable. Until now: the
complaint procedure has not been used.
Research evaluation & EVC audit
A similar procedure as presented above applies for the research evaluations.
Regarding the EVC-audits, a complaint procedure is in place. This is laid down in the
NQA protocol. Complaints of candidates will be forwarded to the EVC-provider and the
NKC-EVC.
Evaluation
NQA has procedures in place for processing appeals and complaints. The procedures
are checked within NQA’s ISO 9001 certification and are found to comply with its
requirements. NQA regrets that in three assessment procedures it came to an appeal
procedure. Although it is confident that the outcomes of the procedures will/have
prove(n) that our method of working complies with the standards for external quality
assurance. Furthermore, NQA is pleased that no complaints have been presented. In
addition, NQA wants to emphasize that also no appeals and/or complaints regarding
our method of working have been addressed to NQA through other organisations, like
the NVAO.
55
56
9. Information and opinion of stakeholders
As described in chapter 6 and in paragraph 7.6, NQA organises several evaluative
activities that provide us with information and the opinion of our main stakeholder. In
this chapter you will find a summarized overview of the results of our (main)
evaluations. The chapter addresses the outcomes of the evaluations held under
assessed degree programmes, our account evaluations, our consultations with the
NVAO and the yearly ISO (9001) audit.
9.1 Evaluation of degree programme assessments In 2015 the evaluations held under assessed degree programmes show that they were
(very) satisfied with the assessments conducted by NQA. Expressed in a score (1 –
10) the programmes awarded NQA with a 7,7 (average score). The cooperation with
the NQA project manager is highly appreciated. They are considered: flexible,
accesible, open, constructive and clear. Furthermore, the feedback shows our project
managers are audit experts that value and live up to agreed upon deadlines. Similar
positive feedback was given on the evaluation items concerning the members of the
panel, the site visit and the report. To enhance the quality of our assessments/service
remarks regarding the NQA portal (for the upload of the required documents), the
required documents itself (reduction of overlap), the capability of chairs of the panels
to present the main conclusions at the end of the site visit and the balance in the
assessments between accountability and improvement.
As the return rate of evaluation forms in 2016 was low (considered a point of
attention for NQA), the results of 2016 and 2017 were combined. In total 20
evaluation forms were returned in the period January 2016 – November 2017.
Although, NQA has some reservations because of the return rate, it seems that degree
programmes have become more satisfied with our assessments. The satisfaction,
expressed on a 1-10 scale, climbed to an average of 8,3. NQA is very pleased with
this result; it shows NQA manages to carry out valuable assessments. Where
programmes show constructive criticism on (aspects of) the assessments, they also
make clear that this is sometimes beyond our span of control. For example when a
project manager or a panel member gets sick and has to be replaced. More valuable
to us is the feedback that we can be more precise in preventing typing errors and/or
grammatical errors. And even more important is the feedback that we could
strengthen the ‘feel of the assessment during the site visit’ with the ‘motivations laid
down in the report’.
57
9.2 Account evaluations
The evaluations by the account managers show that the institutions are very satisfied
with the assignments conducted by NQA (assessments of degree programmes and
research evaluations). In the consultations institutions let us know that they are
pleased with our procedures, methodologies as well as with the execution and the
evaluation of the projects. As the evaluations by the account managers are on a more
strategic level, this is also a platform to discuss the system of accreditation in a
broader sense. This shows us that our position is valued and that it is appreciated that
NQA is a constructive critical partner in this system. Our continuous pleads for
independency of peer panels and for ensuring valid assessments, is appreciated. The
account evaluations also address the assessments/evaluations of degree programmes
that were less pleased with the outcomes. But, the consultations on the level of the
institutions (accounts) shows that we manage to overcome these issues with respect
to the independency of the expert panel and its judgement.
The account evaluation also provides the opportunity to look forward to the
assessments to come. The visitation schedule for 2018 shows a large peak, which
means that a lot of degree programmes have to be assessed. NQA is not sure it can
hold on to the tight deadlines we realized in the past few years. This topic, as a result
of the imbalance in the visitation schedule, will be addressed in the account
evaluations.
9.3 Consultations with the NVAO
The consultations with the NVAO show their appreciation for our work as an
assessment agency. In general the NVAO is pleased with our contribution in the
system of external quality assurance. Regarding our working methods, our
performances on panel composition are considered a strong point. This concerns
specifically our focus on expertise, independency and our time management within
this process. The NVAO is also satisfied with our reports, which in (far) most cases are
accepted without any further questions. Lately, the motivation of panels to judge a
programme as ‘good’ has become a point of attention of the NVAO. This will be
addressed in our consultation in December.
9.4 ISO audit (9001)
NQA is an ISO member since 2006. In a yearly audit NQA shows its compliance with
the ISO-standards (9001). The outcomes are positive, year after year. Our level of
control was awarded a four in 2014 and 2016. In 2015 NQA was awarded a 4,5 (all on
58
a scale of 5). This made us particularly proud, because the audit in 2015 concerned
the tri-annual re-certification audit. The following aspects are pointed out in the ISO-
reports15 as our strong points:
- NQA shows clear, continual quality improvement (evaluations, assessments,
analyses, approval rate panel composition, analyses judgements of peer panels
(on national level);
- Calibration of criteria with other assessment agency;
- High client/customer satisfaction;
- Co-reading of draft reports by a NQA colleague;
- Proper evaluations available and follow-up (improvement measures) appointed.
- Intervision between NQA auditors (project managers);
- Evaluation panel members;
- The administrative process is robust;
- Professional development (training/instruction) of panel members and project
members contribute to the quality of their (assessment) judgements.
Next tot these positive remarks, the ISO audits led to a few observations or
recommendations. NQA took these in consideration, which led to several improvement
measures. For example:
- Results of evaluations are presented in an overview to improve process to come
to improvement measures;
- The visibility of the evaluation of other services was strengthened;
Another observation concerned the connection between the internal audits of NQA to
our own standards for processes. This will be improved before our next ISO-audit in
Q1 2018.
15
ISO-reports: 1-13-2015, 10-14-2015 & 12-8-2016
59
10. Current challenges and areas for further development
Writing this self-evaluation, including a SWOT-analysis (paragraph 3.4) and combined
with our regular evaluative activities, lead to several current challenges and areas for
further development. Amongst others, NQA considers the following aspects by all
means relevant:
- NQA wants to continue its pleads for a ‘leveled playing field’ between the
assessments agencies, especially with regard to the independence and
expertise for panel members.
- NQA believes and has proven to conduct its assessment activities (assessment
of degree programmes, research evaluations and EVC audits) based on the
relevant and national approved frameworks. Regarding the ESG standards it
feels its compliance is sufficient, but could be strengthened. Especially
regarding ESG 2.6, ESG 3.4. Although this also means, that NQA has to move
other parties to act.
- In one or two months an evaluation by the Inspectorate of Education will be
published on the system of accreditation. This could influence our position
within the system and/or influence our working methods.
- Presented as opportunities in the SWOT-analysis: NQA wants to achieve a light
grow of its portfolio of research evaluations. It also wants to explore
possibilities to contribute to the quality culture within institutions/degree
programmes and explore products that show that degree programmes are
permanently accreditation-proof.
- A challenge for 2018 (second half) is the amount of degree programmes that
has to be assessed. The visitation schedule shows a high peak for this period.
Many degree programmes already made arrangements with NQA for that
period. The amount is so overwhelming that NQA considers to slightly loosen its
well appreciated, tight deadlines for the report phase of the projects.
