ii

CONTENTS

Page

LIST OF MOST COMMONLY USED ABBREVIATIONS iv

FOREWORD vi

1. INTRODUCTION 1

2. INTERNAL ASSESSMENT AS A CORE PROCESS IN QUALITY ASSURANCE 2

3. THE EUROPEAN REFERENCE FRAMEWORK 7

4. INTERNAL QUALITY ASSURANCE SYSTEMS WITHIN THE NATIONAL JURIDICAL FRAMEWORK 12

5. EUROPEAN TRENDS WITHIN THE FIELD OF QUALITY ASSURANCE 17

5.1 The European dimension of quality assurance 17 5.2 The influence of the Bologna Process 19 5.3 The dynamics of external quality assurance 21

5.4 From quality assurance to quality enhancement 26 5.5 Future directions and challenges 30

6. INTERNAL QUALITY ASSURANCE SYSTEMS – THE EUROPEAN PANORAMA 33

6.1 The implantation of internal quality assurance systems 33

6.2 The valorisation of internal quality assurance systems 36

6.3 Auditing internal quality assurance systems 40

7. INSTITUTIONAL AUDIT PROCESSES IN EUROPE 43

7.1 Institutional audit processes – Case studies 43

7.1.1 ‘Academic Infrastructure’ – External standards for quality assurance in the United Kingdom 43

7.1.2 ‘Quality Enhancement Framework’ – A radical approach to quality in Scotland 48

7.1.3 ‘Quality Reform’ – The Norwegian system 51

iii

7.1.4 Quality Network – The participation of the universities in the Swiss model 55

7.1.5 Postponement of audits – A backward step in Denmark 57 7.1.6 Respect for Autonomy – The Finnish model 59

7.1.7 ‘System Accreditation’ – A new model for accreditation in Germany 62

7.1.8 Conciliating support and audit– The Austrian Quality Assurance Agency (AQA) 69

7.1.9 Earning trust – The case of the Netherlands and Flanders 73 7.1.10 The AUDIT programme – Spain 76

7.2 Core characteristics of the institutional audit processes – A comparative analysis 84

7.2.1 Scope and motivation behind audit processes 84 7.2.2 Criteria and methodology 87 7.2.3 Consequences of audit processes 90 7.2.4 Follow-up 92 7.2.5 Summary table 92

8. CONCLUSIONS 95

8.1 Specification of internal quality assurance systems 95

8.1.1 Proposal of a frame of reference for internal quality assurance systems 98

8.2 Core elements for the certification of internal quality assurance systems 103

8.3 Suggestions for future work 107

REFERENCES 110

ANNEX – PART 1 OF THE EUROPEAN STANDARDS AND GUIDELINES 117

iv

LIST OF MOST COMMONLY USED ABBREVIATIONS

A3ES Agência de Avaliação e Acreditação do Ensino Superior (Portugal)

ACQUIN Accreditation, Certification and Quality Assurance Institute (Germany)

ACSUG Axencia para a Calidade do Sistema Universitario de Galicia

ADISPOR Associação das Instituições Superiores Politécnicas Portuguesas

AERES Agence d’Evaluation de la Recherche et de l’Enseignement Supérieur (France)

AHELO Assessing Higher Education Learning Outcomes (OCDE)

ANECA Agencia Nacional de Evaluación de la Calidad y Acreditación (Spain)

APESP Associação Portuguesa do Ensino Superior Privado

AQA Austrian Agency for Quality Assurance

AQAS Agency for Quality Assurance by Accreditation of Study Programmes (Germany)

AQU Agência per a la Qualitat del Sistema Universitari de Catalunya

BFUG Bologna Follow-up Group

CNAVES Conselho Nacional de Avaliação do Ensino Superior

CNE Comité National d’Evaluation (France)

CRE Association of European Universities

CRUP Conselho de Reitores das Universidades Portuguesas

EHEA European Higher Education Area

ELIR Enhancement-led Institutional Review (Scotland)

ENQA European Association for Quality Assurance in Higher Education

EQAR European Quality Assurance Register

ESG Standards and Guidelines for Quality Assurance in the European Higher Education Area

ESIB National Unions of Students in Europe

v

ESU European Students’ Union

EUA European University Association

EURASHE European Association of Institutions in Higher Education

EVA Danish Evaluation Institute

FINHEEC Finish Higher Education Evaluation Council

FUP Fundação das Universidades Portuguesas

GAC German Accreditation Council

HEFCE Higher Education Funding Council – England

HEI Higher education institutions

HRK Hochsschulrektorenkonferenz (German Rectors’ Conference)

KMK Kultusminister Konferenz (Conference of Ministers for Education and Culture – Germany)

NOKUT Norwegian Agency for Quality Assurance in Education

NVAO Accreditation Organisation of the Netherlands and Flanders

OAQ Centre of Accreditation and Quality Assurance of the Swiss Universities

QAA Quality Assurance Agency for Higher Education (United Kingdom)

QAHECA Quality Assurance for the Higher Education Change Agenda Project (EUA)

QEF Quality Enhancement Framework (Scotland)

RJIES Regime jurídico das instituições de ensino superior

SFC Scottish Funding Council

SUC Swiss University Conference

TQA Teaching Quality Assessment (United Kingdom)

ZEvA Central Evaluation and Accreditation Agency Hanover (Germany)

vi

FOREWARD

The mission of the Agência de Avaliação e Acreditação do Ensino Superior (A3ES) is to assure the quality of higher education in Portugal, through evaluation and accreditation of its study cycles and through other types of evaluation of a scientific nature, and thus to ensure Portugal’s integration in the European higher education quality assurance system.

The A3ES is conscious that its action must be underpinned by research and a continuous effort towards improvement. In order to fulfill these objectives, a research and analysis group was set up within A3ES, which is responsible for data collection, information processing, analytical studies and other types of research, within the area of quality assurance in higher education.

Within the terms of the legal Framework which instituted the current system of evaluation and accreditation in higher education, the essential task to be undertaken by A3ES is the promotion and dissemination of a culture of quality within higher education institutions. In order to do this, A3ES will publish a series of studies on themes linked to the quality of higher education systems, which will serve as a basis for permanent discussion with Higher Education Institutions and to make available information concerning the methods and procedures adopted by the agency in the course of its activity.

Within the terms of Law 38/2007, it is the responsibility of higher education institutions to adopt quality assurance policies and procedures for their implementation; to develop a culture of quality and of quality assurance, and to promote and implement a strategy for continuous enhancement of quality. The same Law determines that external evaluation procedures “should take into account the efficacy of the internal quality assurance procedures”. Moreover, in the preamble to Decree-Law 369/2007, one of the mainstays of the new system of evaluation and accreditation is defined as “The demand for the development and implementation by higher education institutions of their own quality assurance systems, which may be subject to certification”

The legislation thus reaffirms that the quality of teaching is primarily the responsibility of each higher education institution, and that they must create internal structures and procedures to assure this quality. It will be the responsibility of A3ES to support the institutions in the implementation of their internal quality assurance systems and to undertake audits with a view to certification of the internal quality procedures of the institutions.

vii

The first publication by A3ES is, very appropriately, a study of European Processes for Assessment and Certification of Internal Quality Assurance Systems, by Professor Sérgio Machado dos Santos, whose expertise in the area is extremely well known. The aim of this study is to make a comparative analysis of internal quality assurance systems implemented in European countries of reference, so as to provide institutions with a basis on which to establish of their own internal systems. It will also underpin the work of supporting the institutions that A3ES must undertake in the ambit of its competences.

1. INTRODUCTION

The A3ES, in its objectives and activity plans, foresees the establishment of institutional audit mechanisms leading to the possibility of certification of internal quality assurance systems within national higher education institutions, as an essential tool for a subsequent simplification of procedures in the external assessment and accreditation processes which apply to institutions and programmes. The priority task for the Agency is, however, by legal requirement, to promote the accreditation, in the short term, of all the degree programmes that are already operating in all higher education institutions, together with the ex-ante accreditation of new study cycles. Therefore, preparation of the audit mechanisms will only start in 2010, for implementation in 2011/2012.

This study corresponds to a preliminary phase of the process, the aim of which is to gather data on the subject, which is relatively new to Portugal, taking advantage of the experience of other countries in which some of the audit mechanisms foreseen are already in use or are in the late stages of development.

The specific aims of the study are: - to establish the importance of internal assessment within the framework of

quality assurance, in the light of the European guidelines for this area; - to analyse the national legal context, with regard to internal quality assurance

systems and their certification; - to analyse the main European trends in the area of specification and

certification of internal quality assurance systems, in order to identify and characterise cases of good practice;

- to identify core components for the certification of internal quality assurance systems which may steer national higher education institutions, without losing sight, however, of the flexibility necessary for the desirable development of innovative models which may arise within the ambit of the responsible exercise of institutional autonomy.

In the final chapter, the study includes a concrete proposal for a frame of reference for internal quality assurance systems in Portuguese higher education institutions, along with suggested guidelines for the design and development of the institutional audit process to be adopted by the A3ES.

These proposals were put to public debate throughout the first semester of 2010 through their presentation to, and discussion by, the representative bodies of the different sectors of higher education, and also in a number of higher education institutions, whenever such was requested.

2

2. INTERNAL ASSESSMENT AS A CORE PROCESS IN QUALITY ASSURANCE

Among the many effects of the radical changes that occurred in the context of higher education in the last quarter of the 20th century was the emergence of serious concern with quality assurance, on the part of both the institutions themselves and society in general, placing assessment incontrovertibly on the agenda of higher education.

Among the changes that occurred, some particularly relevant aspects were:

a) The massification of access, relating to the exponential growth in higher education, the diversification of provision of undergraduate and postgraduate programmes, the diversification of institutions as a consequence of the fast development of the polytechnic and private sectors, as well as the new expectations of the public with regard to higher education as a source of training and education, not just for elites but mainly as a means for the exercise of active citizenship in the context of the knowledge society.

b) Internationalization, namely with regard to the need to confer international validity on the certification of qualifications, in order to facilitate the transferability of academic and professional qualifications and consequently the continuation of studies and the access to a profession, in face of an increased mobility of students and graduates. This question is particularly relevant in the context of the European Union, as a fundamental condition for ensuring the right to free circulation and thus contributing to the full exercise of European citizenship.

c) Greater awareness of their rights on the part of citizens, and the subsequent demand for quality.

As a consequence of these environmental changes in higher education, institutions were faced with new challenges and expectations, namely the problem of how to preserve quality in the face of massive, rapid and sometimes uncontrolled growth, which made it necessary to consider quality from a more institutional perspective. Indeed, with great numbers of students, and a rapid increase in the teaching staff, there tends to be some dilution of the traditionally high level university standards and this no longer suffices to implicitly ensure the quality that is inherent to higher education. At the same time, when faced not only with a variety of supply, but also with marketing strategies which are sometimes aggressive, society and applicants for higher education – together with their families and future employers – feel the need for better information

3

about the quality of the programmes on offer, which has led governments to take measures of “consumer protection”.

In part due to such concerns, quality assurance systems began to be implemented, using different approaches according to their geographical location, particularly in terms of the so-called “evaluation processes”, which predominated in Western Europe, and the “accreditation processes” which were developed in Eastern Europe, for example, where they took advantage of the vast experience of the United States of America in this field.

It is important to note that evaluation and accreditation have complementary objectives and that both are of great social interest. Indeed, evaluation, which aims to monitor and constantly improve quality, represents what could be considered as the more noble function and with greater impact on society, which is to promote the quality of teaching, research, cultural action, and interaction with society. Accreditation, on the other hand, aims to guarantee compliance with the minimum requirements necessary for a programme or institution to gain official recognition as such.

The initial duality between the evaluation and accreditation processes has gradually become more blurred and nowadays it is generally agreed that evaluation and accreditation are inseparable and are, indeed, two sides of the same coin. It would certainly make little sense that the process of evaluation – while transmitting to the public a message concerning the quality of an institution or programme –, should not objectively identify cases in which the minimum requirements of quality necessary for accreditation are not fulfilled. On the other hand, accreditation alone provides society with important information, but which is insufficient because a programme may respect the minimum requirements in order to be officially recognized, but may only be reasonable or mediocre in terms of quality.

Currently, therefore, quality assurance systems tend to combine different complementary aims, with more or less focus on each feature depending on whether the national external quality assurance systems are more concerned with a process of control and certification of conformity and verification associated with the dimension of accreditation or more concerned with transparency and improvement, which has more to do with the dimension of evaluation, focusing on the production of validated, objective and intelligible information, and on the systematic and sustained improvement of quality within the institution.

Another dichotomy regarding quality assurance concerns internal and external assessment, two facets of the system which are also complementary. Here, it is important to keep in mind the idea – which is today consensual – that quality and

4

quality assurance are primarily the responsibility of the higher education institutions themselves. Indeed, it is within the institutions that the battle for quality will be won or lost.

This was the position of Portuguese state universities (including UCP and ISCTE) when, in 1993, they took the initiative of launching an assessment programme, in order to trigger an internal culture of quality, which other higher education institutions later adhered to. In the document of guidelines1 which largely shaped the law on assessment passed in 19942, there is emphasis on the level of responsibility of institutions with regard to the community “which implied continual updating of teaching, with sustained improvement and the ability to respond to the great challenges of our time”, and it advocated “the absolute need for each higher education establishment to institutionalize an internal system of self-assessment based on previously defined indicators”. The success of this initiative in terms of the impact it had within the institutions was made very clear in the report on the self-assessment of the Portuguese system of quality assurance, prepared in the ambit of the assessment promoted by ENQA. Among the conclusions of this report one of the strengths of the system was identified as:

“the good acceptance and enthusiasm by the part of many institutions, which have assumed their participation in the evaluation process as an opportunity to reflect on their organisation and activities, and to take up quality culture and quality assurance as strategic elements for their institutional development”,

noting, consequently, “the positive movement towards the institutionalization of internal quality assurance procedures” 3.

This is also the position that is being taken at European level throughout the Bologna Process, expressed namely in the Berlin Communiqué4, in which Ministers responsible for higher education in signatory states to the Bologna Declaration explicitly stated that:

“Consistent with the principle of institutional autonomy, the primary responsibility for quality assurance in higher education lies with each institution itself and this provides the basis for real accountability of the academic system within the national quality framework”.

1 CRUP (2003). Guiding Principles for the Evaluation and Follow-up of Activities within Higher Education Institutions (translated title). 2 Law nº 38/94, 21st November. 3 Santos, S.M. (Rapporteur) et al. (2006). Review of the Quality Assurance and Accreditation Policies and Practices in the Portuguese Higher Education – Self-Evaluation Report, p. 78. 4 Realising the European Higher Education Area (2003). Communiqué of the Conference of Ministers responsible for Higher Education, Berlin.

5

This declaration is reflected in the document Standards and Guidelines for Quality Assurance in Higher Education, adopted in 2005 at the Bergen Ministerial Meeting, which among its underlying basic principles includes5:

- “Providers of higher education have the primary responsibility for the quality of their provision and its assurance”;

- “There should be encouragement of a culture of quality within higher education institutions”,

and, in accordance with these principles, it gives special relevance to mechanisms of internal quality assurance, as will be seen in more detail further on.

It is patently obvious from this line of thought that internal quality assurance structures and mechanisms must be a basic first line of intervention in assessment procedures, with the ultimate aim of promoting the interiorisation of the need for a culture of quality which permeates all activities undertaken within the institution. The following points also emerge as fundamental in explaining the idea underlying internal quality assurance:

a) More than a bureaucratic exercise of control and certification of conformity with external orientations, internal assessment is essentially a permanent process aiming at enhanced quality. As such, it involves monitoring and control processes, but also reflection and subsequent intervention. It presupposes systematic monitoring of various activities, data collection and the construction of indicators. In other words, it includes a dimension of mensurability, which is essential for the credibility of the assessment process, for the establishment of benchmarks and, above all, as a starting point for subsequent reflection on the information gathered during monitoring, from which conclusions should be drawn and consequences should be expressed in proposals of measures for adaptations or corrections to be made. That is, assessment involves a process of retroaction for quality enhancement.

b) The process of internal assessment requires that all internal activities should be assessed (teaching at undergraduate and postgraduate level, research, community outreach, as well as all of the actors involved). This assessment should occur naturally, as part of everyday activities, by means of simple, agile mechanisms, which at the same time are effective and produce outcomes.

c) Internal assessment should actively involve all relevant actors, in a process implying collective responsibility, as something that is meaningful to and

5 ENQA(2005). Standards and Guidelines for Quality Assurance in the European Higher Education Area, p. 13.

6

shared by everyone. In other words, it should induce a sense of belonging, ownership and co-responsibility among the academic community and not merely be seen as an imposition on the part of the academic management or political power.

d) The process of internal assessment should obey an institutional policy for quality assurance and should follow procedures which have been duly institutionalised. It is not, therefore, compatible with ad hoc procedures; in other words, it requires pre-defined measures and appropriate support structures.

These points concerning the main function of internal assessment as part of the quality assurance systems do not diminish the importance of external quality assurance, which is not only necessary but also essential both to validate internal assessment mechanisms – which must be included in the review to be undertaken by the external assessors – but also as a source of information and an independently validated judgement.

It has been repeatedly pointed out in various documents pertaining to the Bologna Process that higher education is a public good and a public responsibility. Public authorities and society in general cannot, therefore, distance themselves from this public good and even less from its quality. External assessment must thus be seen in the scope of the social dimension of higher education, addressing, in a complementary way, both of the above-mentioned dimensions: on one hand, as a critical view on the quality and quality assurance within the institutions, and on the other, as validation of objective information presented in a manner which is easily understood by society.

7

3. THE EUROPEAN REFERENCE FRAMEWORK

The systems of quality assurance are one of the essential mainstays for the implementation of the aims of the Bologna Declaration. Indeed, they make a decisive contribution to the mutual recognition of degrees and study periods, and the resulting transferability of academic and professional qualifications, which, as has already been pointed out, is a crucial condition for the mobility of European citizens

The Bologna Declaration, however, says little about quality assurance, dedicating only one paragraph to the subject, in which it advocates the promotion of European co-operation in quality assurance with a view to developing comparable criteria and methodologies6. The importance of assessment was thus implicitly taken for granted and the development of assessment systems was left up to each country, within the principle of subsidiarity.

However, some protested that assessment was the absent element in the Bologna Declaration and they insisted on more precise objectives regarding quality assurance. Perhaps because of this, but also as a consequence of the enlargement of the EU to Eastern European countries and due to concerns regarding transnational education, the Prague Communiqué7, in 2001, took up the theme. A specific section was dedicated to it, broadening the concept by establishing, as an objective within the domain of quality assurance, the development of the conditions for the mutual recognition of evaluation and accreditation/certification mechanisms. An appeal was also made to higher education institutions, national agencies and to the ENQA to work together in the establishment of a common frame of reference and in the dissemination of information about good practice.

It was, nonetheless, at the Berlin meeting of 2003 that a most proactive position was taken when, with a view to accelerating the Bologna Process, Ministers defined intermediate priorities in three areas of action: quality assurance, the two-cycle system and the recognition of degrees and periods of study. Returning to the need for the development of mutually acceptable criteria and methods for quality assurance, the Berlin Communiqué established some parameters to be observed in national systems of quality assurance8 and delegated to the ENQA – which was still known as the European

6 The European Higher Education Area (1999). Joint Declaration of the European Ministers of Education Convened in Bologna on the 19th of June 1999. 7 Towards the European Higher Education Area (2001). Communiqué of the meeting of European Ministers in charge of Higher Education in Prague on May 19th 2001. 8 In the Berlin Communiqué it is stated that: “[The Ministers] agree that by 2005 national quality assurance systems should include: a definition of the responsibilities of the bodies and institutions involved; evaluation of programmes or institutions, including internal assessment, external review,

8

Network for Quality Assurance in Higher Education, whose membership included representatives of national and regional evaluation agencies and government representatives9 – the responsibility of preparing, through its associates and in cooperation with the European University Association (EUA), the European Association of Institutions in Higher Education (EURASHE) and the student association National Unions of Students in Europe (ESIB)10, a commonly acceptable set of standards, procedures and guidelines on quality assurance and to explore ways of ensuring an adequate peer review system for quality assurance and/or accreditation agencies or bodies. This work was undertaken between 2003 and 2005 by ENQA in close cooperation with the E4 group11, which was made up of the above-mentioned associations, with the participation of the European Commission. It led to the document Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG).

At the 2005 Bergen meeting, the Ministers approved this document and committed themselves to introducing into their respective national systems of quality assurance the guidelines and criteria established within it. The Bergen Communiqué12 also welcomed the idea of the creation of a European register of quality assurance agencies, requesting ENQA to develop the practicalities of implementation of this register in cooperation with EUA, EURASHE and ESIB. The importance of cooperation between national agencies was also underlined, with a view to enhancing mutual recognition of decisions relating to accreditation or assessment procedures, and an appeal was made to higher education institutions to continue their efforts to enhance the quality of their activities through the systematic introduction of internal mechanisms and their direct correlation to external quality assurance.

In 2007, The London Communiqué13 returned to the theme, emphasising the progress made so far: participation of students and the publication of results; a system of accreditation, certification or comparable procedures; international participation, co-operation and networking”. 9 Later, at the General Assembly of 4/11/2004, the ENQA became an association, calling itself the European Association for Quality Assurance in Higher Education, only including as full members the associated agencies. Representatives of national governments attend the meetings of the General Assembly as observers. 10 In May, 2007 the ESIB changed its name to ESU – European Students’ Union. 11The E4 Group, which included ENQA, the ESIB (now, ESU), the EUA and the EURASHE, began to meet regularly in 2001 with a view to contributing to the creation of a European dimension for quality assurance. It played a decisive role in the preparation of the ESG and in the design and implementation of the EQAR. 12 The European Higher Education Area – Achieving the Goals (2005). Communiqué of the Conference of European Ministers Responsible for Higher Education, Bergen. 13 Towards the European Higher Education Area: responding to challenges in a globalised world (2007). London Communiqué.

9

“The Standards and Guidelines for Quality Assurance in the EHEA adopted in Bergen (ESG) have been a powerful driver of change in relation to quality assurance. All countries have started to implement them and some have made substantial progress. External quality assurance in particular is much better developed than before. The extent of student involvement at all levels has increased since 2005, although improvement is still necessary. Since the main responsibility for quality lies with HEIs, they should continue to develop their systems of quality assurance. We acknowledge the progress made with regard to mutual recognition of accreditation and quality assurance decisions, and encourage continued international cooperation amongst quality assurance agencies”.

The Ministers asked the E4 Group to organize an annual European Forum on Quality Assurance14 to facilitate the exchange of information on good practice and to guarantee continued enhancement of higher education in the European space. They agreed to the implementation of the European Quality Assurance Register (EQAR), along the lines of the model proposed by the E4 Group, namely with regard to voluntary adhesion on the part of agencies, to its independent nature, its transparency and the fact that it should be self-financed. It is the intention of EQAR to encourage registration of quality assurance agencies which are operating within the European Higher Education Area in conformity with the ESG. This body was formalized in March, 2008, and included as actors the E4 Group and social partners, including government representatives as observers. In a recent report on the main developments of the Bologna Process, the decision taken in London to create EQAR was considered a historic landmark as it corresponded to the establishment of the first legal body to be set up through the Bologna Process15.

The Louvain Communiqué16, which was already quite focused on the extension of the Bologna Process beyond 2009, did not add anything of substance to the issues discussed above, but it included, among its priorities and challenges for the next decade, the development of “multidimensional transparency tools” which in practice opened up the door to the establishment of rankings among European higher education institutions17. The E4 group was asked to continue their cooperation in further developing the European dimension of quality assurance, and in particular to ensure that the EQAR is evaluated externally, taking into account the views of the stakeholders.

14A first European Forum on Quality Assurance had been held in 2006, in Munich, organized by the EUA in partnership with the other institutions which make up the E4 Group, which proved to be a great opportunity for discussion of the developments underway in various countries. This was the initiative behind the recommendation of the Ministers that this type of meeting should become an annual event. 15 EC(2009). Higher Education in Europe 2009: Developments in the Bologna Process, p. 16. 16 The Bologna Process 2020 – The European Higher Education Area in the new decade (2009). Communiqué of the Conference of European Ministers Responsible for Higher Education, Leuven and Louvain-la-Neuve. 17This reference, made in the Louvain Communiqué, to instruments aimed at more detailed information on institutions, allegedly to make the diversity of the European Higher Education Space more transparent, met with the strong disagreement of students, but strangely did not provoke a visible reaction on the part of organisations representing higher education institutions.

10

The document on standards and guidelines adopted in Bergen became the main European reference for assessment of higher education, and has naturally conditioned the national quality assurance systems. It is organized in three sections regarding, respectively, internal quality assurance in higher education institutions, external assessment of higher education and quality assurance within the agencies themselves. The standards do not aim to be prescriptive, and in general they do not go into detail concerning procedures, in order not to impose unique models and consequently to provide leeway for the exercise of autonomy by the institutions and by the agencies themselves. Indeed, it is explicitly assumed that, as an underlying principle, the document on standards and guidelines endorses the spirit of the EUA Graz Declaration, where the idea is defended that “the purpose of a European dimension to quality assurance is to promote mutual trust and improve transparency while respecting the diversity of national contexts and subject areas”18.

With regard to internal assessment, which is the main focus of this study, seven standards are defined, in which the previously mentioned principles for internal assessment are clearly in evidence, and which can be organized in three different areas of concern:

a) The first norm deals in detail with the question of institutionalization of internal quality assurance, establishing that higher education institutions must define a policy for quality assurance and also the associated procedures it involves, including a clear definition of the standards of its programmes and degrees They should also make a explicit effort to develop a culture which recognizes the importance of quality and of quality assurance for its activities. With these objectives in mind, they should at the same time develop and implement a strategy for the continuous enhancement of quality. The norm adds that the adopted strategy, policy and procedures should have a formal status, be publicly available, and should incorporate an appropriate role for students and other stakeholders.

b) The next four norms, relating to the most pertinent areas for the assessment of teaching, deal with, respectively, the educational offer and operation of the programmes, the assessment of the students, the teaching staff and the resources. They establish that: - higher education institutions must have formal mechanisms for the

approval, periodic review and monitoring of their programmes and degrees;

18 EUA (2003). Graz Declaration 2003 – Forward from Berlin: the Role of the Universities, p. 9.

11

- students should be assessed using published criteria, regulations and procedures which are applied consistently;

- institutions must have means of ascertaining that their teaching staff is appropriately qualified and competent to teach, and these procedures must be made public;

- institutions must guarantee that the resources available for the support of student learning are adequate and appropriate for each programme offered.

c) The last two norms concern data collection and its use and publication, and specify that: - higher education institutions should develop a system for collection,

analysis and utilization of relevant information for the effective management of their programmes of study and other activities;

- institutions should regularly publish updated, impartial, and objective information, both quantitative and qualitative, about the programmes and degrees they are offering.

The importance given to internal quality assurance systems is reinforced by the fact that the first norm in the second part, regarding external assessment, makes it clear that “external quality assurance procedures should take into account the effectiveness of the internal quality assurance processes described in Part 1 of the European Standards and Guidelines”.

In spite of the relative detail with which the standards are formulated, each one is accompanied by a set of guidelines which justify, clarify and look in detail at the norm, even suggesting possible pathways for its implementation. Thus, this document is not only a crucial reference for the specification of internal quality assurance systems, but also a source of invaluable help for their design and implementation.

In a final note on the European framework described above, it is important to mention that the document approved in Bergen explicitly states that it only refers to the dimension of teaching and learning: “The standards and guidelines relate only to the three cycles of higher education described in the Bologna Declaration and are not intended to cover the area of research or general institutional management”19.

19 ENQA(2005). Standards and Guidelines for Quality Assurance in the European Higher Education Area, p. 11.

12

4. INTERNAL QUALITY ASSURANCE SYSTEMS IN THE NATIONAL JURIDICAL FRAMEWORK

The demand for appropriate assessment of the quality of teaching is foreseen in the Constitution of the Portuguese Republic, in Art 76º relating to the autonomy of universities, which after the constitutional revision of 1997 states: “In the terms of the law, the universities enjoy statutory, scientific, pedagogical, administrative, and financial autonomy without detriment to the proper evaluation of the quality of the teaching”. This constitutional principle reinforces the idea underlying the binomial autonomy/responsibility, in which assessment is undoubtedly seen as a fundamental pillar for the full exercise of institutional autonomy.

The RJIES – the Portuguese juridical regime governing higher education institutions – (Law nº 62/2007, 10th September) reiterates this principle by establishing in Art. 11, nº 5, that “the autonomy of Higher Education Institutions does not preclude (…) accreditation and external assessment, in the terms of the law”.

Internal assessment is dealt with in Art.147, nº 1, of the RJIES: “Within the terms of their statutes, higher education institutions should establish mechanisms for regular self-assessment of their performance”. The legal imperative is thus established for institutions to include in their statutes the way in which their respective internal quality assurance systems will be organised. This statute dignifies and confers a higher degree of institutionalization on internal assessment, which was not previously the case, as up until the present juridical regime was established “only rarely did any university or polytechnic statutes, even those approved after the Assessment Law (Law nº 38/94, 21st November), include any structure specifically responsible for overseeing quality, although in the Constitution this was considered to be interconnected with autonomy”20.

The questions of transparency, information and dissemination, which in the terms of the European standards are essential to internal quality assurance systems, are dealt with in detail in Arts. 161 and 162 of the RJIES, clearly geared towards “consumer protection”:

“Art. 161 - Transparency

1. Higher Education Institutions must make available on their Internet site all the information necessary for easy understanding of the cycles of studies offered and the degrees awarded, research undertaken, and services offered by the institution. 2. This information must include reports on the self-assessment and external assessment of the institution and of its organs, as well as on its study cycles .

20 Simão, J.V., Santos, S.M. & Costa, A.A. (2005). Ambition to Excellence – The Bologna Opportunity (translated title), p. 188.

13

Art. 162 - Information and dissemination

1. Higher Education Institutions must make clear in their documents which are aimed at public dissemination, and in all their publicity, the precise contents of the official recognition of public interest, and of the authorisation for the operation of study cycles and the recognition of awards. 2. Accurate and adequate information must be made available on the following points:

a) The mission and aims of the institution; b) Statutes and Regulations; c) Organic units; d) Study cycles offered, degrees awarded and curricular structure; e) Teaching staff and types of contract and of link to the institution; f) Academic assessment regime; g) Accreditation certificates and results of the assessment of the institution and

its study cycles; h) The rights and duties of students, including all fees and other required

payments; i) Social support services; j) Pass, failure and employability rates, relating to study cycles offered; l) Any other data foreseen the law and in the statutes”.

Internal assessment is also dealt with in Law nº38/2007, 16th August, which approves the legal framework for the assessment of higher education currently in force. Articles 17 and 18 are particularly important, as is nº 1 of Article 19, which basically transpose into the national juridical regime the European standards which have already been discussed:

“Article 17 - Internal quality assurance

1 - Higher education institutions should:

a) Adopt a policy of quality assurance of their study cycles, according to their mission, and the appropriate procedures for this undertaking;

b) Commit themselves, through concrete measures, to the development of a culture of quality and of quality assurance in their activities;

c) Develop and implement a strategy for continuing quality enhancement.

2 - The strategy, the policy and the procedures referred to above must:

a) Be formally approved by the statutorily legal and competent body of the institution and made public;

b) Ensure the participation of students and other parties interested in the process.

Article 18 – Self-assessment

Within the framework of self-assessment, higher education institutions should:

a) Define formal procedures for approval, follow-up and periodic assessment of its study cycles, which must include:

i) the participation of the pedagogic councils and the opinion of students, namely through the pedagogic councils and the student associations;

ii) the participation of research centres which collaborate in the organisation and operation of study cycles;

14

iii) the participation of external advisory bodies that collaborate with the institution;

b) Adopt appropriate procedures to ensure that the teaching staff possess qualifications and competences necessary to carry out their duties, which must be made available to those responsible for external assessment and must be the object of appraisal in the assessment reports;

c) Ensure that the teaching resources available are sufficient and appropriate for each of the study cycles offered;

d) Ensure that they gather, analyse and use the information necessary for efficient management of their study cycles and other activities;

e) Regularly publish quantitative and qualitative information which is up-to-date, impartial and objective, concerning:

i) the study cycles they offer and the degrees and diplomas they award; ii) keeping track of their graduates for a reasonable period of time, with

regard to employability.

Article 19 – Basic principles of external assessment

1. The procedures for external quality assessment should take into consideration the internal quality assurance procedures referred to in the previous article.”

Decree-law nº 369/2007, which instituted the A3ES, and which, as said in its preamble, concluded “the legislative process related to quality assurance in higher education”, obviously does not go into detail about ways of organizing internal assessment, but includes a very important note in its preamble on the system of assessment, by identifying as one of the axes around which quality assurance systems are organised “the obligation for institutions of higher education to create their own systems of quality assurance, which can be certified”. This is an important point, because it indicates the intention of the lawmakers to foresee the possibility of certification of internal quality assurance systems, although this concept is not dealt with explicitly in the articles of any of the legal diplomas cited.

The action plan for 2009 of the A3ES gives some indications on the importance given to internal quality assurance systems, namely in the following excerpts21:

“By taking into account its specific role in the national higher education quality assurance, the Agency does not ignore that the first responsibility for quality assurance lies with the institutions themselves. This means that the mission of the Agency only makes sense when in permanent dialogue with the institutions and with their interested collaboration. Therefore, a substantial component of the Agency’s activity will be dedicated to supporting the implementation of internal quality assurance systems.”

“It results from the normative framework that created the present assessment and accreditation system that the Agency must assume as its main task the promotion and the diffusion of a quality culture amongst the higher education institutions. Actually, we need to start from the principle that the responsibility for the quality of education lies first of all, with each higher education institution itself that must

21 A3ES (2009). Activity plan for 2009, p. 3.

15

create the internal structures and procedures adequate to promote and assure the quality of that education. The Agency will be responsible for realising audits aiming at certifying the institutional internal quality procedures.”

Thus, the Agency reinforces the idea that certification of internal quality assurance systems is implicitly foreseen in the national juridical framework, concluding, in this regard, after an analysis of the legal principles mentioned earlier: “We thus feel we can use the term ‘certification’ to denote the activities of quality auditing promoted by the agency in order to guarantee the efficiency of the internal institutional mechanisms of quality assurance”.

The Agency draws on these conclusions, which are crucial for the future simplification of the assessment system, and which are to be implemented after the intense work of accreditation of programmes to be undertaken in the short term: “In a stable system, to enter into regular operation at the beginning of the academic year 2011/2012, attention must be focused on internal systems of quality assurance, promoting a lighter system for institutions”22, pointing right from the beginning a possible method for an approach of this kind.

In brief, higher education institutions face the challenging yet rewarding task of implementing their internal quality assurance systems, which are crucial for the consolidation and full exercise of institutional autonomy, and for them to be able to make use of a simplified system of external assessment, with obvious gains in terms of effort and costs involved in individual periodic accreditation of their programmes. A number of institutions have begun to work towards this end in a variety of ways, which is in itself an added value for the system as a result of all the experience that will be accumulated.

However, there are advantages to the establishment of some common guidelines which, without stifling creativity and innovation, will guarantee the fulfilment of some requirements/prerequisites which will allow the above-mentioned certification23.

Here it is worth noting that some polytechnic institutions have decided to apply for certification under the rules established by ISO 9001:2000 and various other university and polytechnic institutions have been certifying some of their support services. The

22 Ibidem, p. 6. 23 ENQA, in ENQA Position Paper on Quality Assurance in the EHEA, in view of the Leuven and Louvain-la-Neuve meeting of ministers responsible for higher education of 28-29 April 2009, addresses the need for a balance between respect for diversity and the adoption of common principles in the folloing terms “The diversity of the EHEA makes a single monolithic and prescriptive approach to quality, standards and quality assurance in higher education inappropriate and impracticable. (…) However, while respecting diversity, [ENQA] does promote the harmonisation and convergence of quality assurance processes, based on common principles”.

16

norms set out in ISO 9001 refer to systems of quality management in service providers, so that they are appropriate for this end – the certification of some particular services within higher education institutions – but to apply them to an institution as a whole, including the organization and operation of teaching, raises difficulties due to the complexity of the teaching/learning processes, which do not fit into the categories provider/client. Efforts have been made to produce guidelines for the application of the ISO 9001 in education, in particular in the documents produced within the framework of the International Workshop Agreement24, which, based on the norms mentioned, promote the adaptation of the terminology, principles and concepts to the reality of educational organizations. It would be worthwhile undertaking a specific study on the effectiveness and real impact of the ISO 9001 on the institutions which adopted them, particularly with regard to an appreciation of the effort made and the level of bureaucracy implied in their application, as well as to their capacity to monitor in depth the teaching /learning processes and allow improvement thereof.

24 IWA 2 (2007). Quality Management Systems – Guidelines for the application of ISO 9001:2000 in education.

17

5. EUROPEAN TRENDS WITHIN THE FIELD OF QUALITY ASSURANCE

5.1 THE EUROPEAN DIMENSION OF QUALITY ASSURANCE

The European dimension of quality assurance in higher education began to emerge at the beginning of the 90’s, following the evolution of the assessment systems in member states and as part of a process of internationalization of assessment. The first steps taken were supranational experiments in evaluating programmes in specific areas, carried out by international teams of experts, with the voluntary participation of institutions. The main objective was a comparison of study programmes in different countries and the mutual learning arising from the process.

Examples of exercises of this nature were: - The International Programme Review of Electrical Engineering, promoted by

the Association of Cooperating Universities in the Netherlands (VSNU) in 1992, regarding a comparative study of the programmes of Electrotechnical Engineering in the Netherlands, Switzerland, Germany, Belgium and the United Kingdom, but which also included the programme operating at the University of Porto;

- The study Business Administration and Economics Study Programmes promoted by the Swedish National Board of Universities and Colleges in 1992, comparing courses in the United Kingdom, the Netherlands, Finland, Germany, France and Sweden;

- A joint project by the Centre for Higher Education Policy Studies (CHEPS) of the University of Twent with the American Accreditation Board for Engineering and Technology (ABET), for comparison of courses of Civil, Mechanical and Chemical Engineering in the Netherlands, Belgium, Switzerland, France Germany and Portugal, in which the Universities of Aveiro and Porto were involved;

- A variety of other undertakings within the ambit of the Erasmus networks and in thematic networks, as a basis for the establishment of mutual confidence likely to increase mobility within these networks.

Thus, these initiatives were embarked upon by the higher education institutions themselves, with a number of different motivations, both internal and external to higher education, in order to develop the international dimension of evaluation.

At the time, internal motivation included increasing student mobility and the consequent problems of transparency, comparability and recognition of periods of study, as well as the exportability of educational programmes, which implied the extending of assessment beyond official systems of higher education. External pressures, on the other hand, were related to the globalization of professions and the corresponding problems of recognition of qualifications, with agreements concerning

18

international commercialization of services and with the action of international organizations interested in the processes of assessment and recognition.

The most significant of these actions, which was the first Europe-wide exercise of this kind, was undertaken by the European Commission in 1994/5 with the pilot project Quality Assessment in the Field of Higher Education, which covered 46 higher education institutions in 17 countries in the European Union and the European Economic Association. The project focused on areas such as Engineering Sciences, Communication/Information Sciences, and Art and Design, and the Portuguese institutions which took part were the Instituto Superior Técnico with its Mechanical Engineering programme and the Instituto Politécnico do Porto with its music programme.

In September, 1998, as a result of the experience gained in this pilot study, the Council of Europe passed a Recommendation on European cooperation in the area of quality assurance in higher education25, so that, as part of their responsibility for the organization of their higher education system, each Member State should establish a transparent quality assessment and quality assurance system. The main objectives were to:

- Assure the quality of higher education, taking account of the specific economic, social and cultural context of each country, while at the same time taking into consideration the European dimension and the existence of a world in rapid evolution;

- Encourage and support higher education institutions to use evaluation as a basis for improving the quality of teaching and learning, as well as training and research;

- Support European and international cooperation, in view of the benefits to be gained from exchange of experience in the above-mentioned areas.

The Recommendation, which specifies principles on which the assessment systems should be based, emphasises the need for autonomy for the agencies responsible for external evaluation, and sets out some standards to be used by the agencies in the course of external evaluation.

There was also a proposal for the creation of a European network to increase cooperation among national assessment systems. This idea was developed by an ad hoc group during 1999, coordinated by the European Commission, and in March 2000 the European Network for Quality Assurance in Higher Education (ENQA) was set up with

25Council of the European Union (1998). Recommendation 98/561/CE, 24th September, 1998, regarding European cooperation, with a view to assuring quality in higher education.

19

the aim of promoting cooperation at European level among all the actors involved in the field of assessment and quality assurance.

Members of ENQA included national and regional quality assurance agencies, Governments through representatives of ministries responsible for higher education, European organizations representing higher education institutions (the Confederation of European Union Rectors’ Conferences, CRE and EURASHE) and the European Commission.

5.2 THE INFLUENCE OF THE BOLOGNA PROCESS

The Recommendation of the Council of Europe and the ensuing creation of the ENQA were a huge qualitative step towards the European dimension of quality assurance, at a particularly significant moment when, in a parallel process arising from an intergovernmental initiative, the Bologna Declaration was signed. Among its concerns with intelligibility, comparability and compatibility of European higher education systems, it called for the “promotion of European co-operation in quality assurance with a view to developing comparable criteria and methodologies”. The ENQA, which was born of an idea previous to the Bologna Process, was naturally seen as the right operational body to attempt to reach this objective.

As a result, the past decade has seen enormous progress in the adoption of quality assurance procedures in European higher education systems, particularly following the Berlin Conference, where, as has been seen previously, the Ministers explicitly declared quality to be a core factor in the establishment of The European Space for Higher Education, and the adoption of the ESG in Bergen two years later, whose principles and guidelines were successively incorporated into national juridical regimes governing higher education. This evolution is patently clear in the data, published in 2008, collected in the survey undertaken by the ENQA, which shows a common pattern, whereby Agencies responsible for external quality assurance were created by an act of law – almost 96% of the agencies were created by a specific law or are recognized by public authorities.26

According to this ENQA survey, the influence of the Bologna Process has become more marked in quality assurance processes, largely due to the efforts to bring them in line with the reference points expressed in the ESG, but also due to the influence of the development of the national qualifications frameworks, which in turn create important frames of reference for the work of the agencies. In a comparison with the survey 26 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 21.

20

published in 200327, it is made clear that one of the most important changes concerns the notion and use of specific criteria and standards which were seen as “emerging as a common feature”, in the various systems, but which in 2008 were actually embedded and in general use.28

A closer look shows that the more immediate effect of the ESG has happened within the higher education institutions themselves, at different rates according to the level of development of the internal quality assurance systems. Thus, in countries where this type of system already existed, the ESG were essentially seen as a point of reference with which to validate the procedures already in use, with regard to how far they fit the European models, in a process that started to gear itself towards the certification of internal systems, as will be seen in more detail in the following chapter. But in cases which were still in the early stages of systematic attention to internal quality assurance, or which were at a phase of transition regarding the types of approach, the ESG served as a useful and revitalizing reference for the design and installation of internal quality assurance systems. However, the main impact of the ESG is being seen at the second and third levels (the organization of the work of the agencies and their meta-assessment), as will be analyzed later.

The use and the influence of the ESG was the main theme of debate at the Second European Quality Assurance Forum. The conclusions of this meeting identified a number of positive effects of the use of the ESG, namely the fact that they provide an operational framework which is appropriate for the development of approaches which in terms of institutional policies and strategies for quality are top down, while also being predominantly bottom up in terms of implementation. This provided a balance between centralization and decentralization of internal systems of quality assurance29. Overall, the opinion is that the ESG are reasonably easy to implement, and that they are able to support creativity, reinforcing the individual nature of each institution, which is why they are a good starting point for new initiatives in the area of quality processes.

There is also mention of the fact that the ESG can facilitate transparency of quality assurance systems, improve their comparability and constitute a potential basis for international cooperation. However, it was noted that in order for the full potential of the ESG to be realized, it will be necessary to continue to invest in exchange of experience and good practices, in clarification of concepts and identification of

27 ENQA (2003). Quality Procedures in European Higher Education – An ENQA Survey. 28 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 85. 29 Harvey, L. (2008). Using the European Standards and Guidelines: Some Concluding Remarks. In EUA(2008), p. 82.

21

obstacles and also in the involvement of stakeholders in their implementation and interpretation.

Potential negative effects were also identified, namely the fear that an overly prescriptive interpretation of the ESG could lead institutions to focus excessively on conformity with standards, to the detriment of taking advantage of good practices already in operation and to the detriment of creativity in the development of internal quality processes. The fact that the ESG refer exclusively to teaching and do not cover other areas such as the assessment of research or the research-teaching relationship was also seen s a significant limitation.

The implementation of the ESG has met with some problems, largely associated with academic conservatism and institutional inertia, which have made it difficult to involve not only teachers but also managers, students and non-teaching staff, in enhancement processes which require significant change in practices and mentalities. In particular, a latent conflict was identified between the practices of the academic staff regarding their own subject area and the more institutional practices, which shapes the majority of quality assurance processes. In order to attenuate these concerns and thus reduce this apprehension it will be necessary to invest in mobilising the internal actors in order to convince them of the value of quality assurance processes.

5.3 THE DYNAMICS OF EXTERNAL QUALITY ASSURANCE

The progress report presented by the Bologna Follow-Up Group (BFUG) at the Ministerial Conference in Louvain30 states that the implementation of the external quality assurance had made significant progress, and that every one of the Bologna Process signatory States had already introduced external quality assurance processes which include the phases of self-evaluation and external evaluation and almost all of them publish the results of the evaluations and implement follow-up measures.

A close examination of the scorecards for 2009 shows that, in regard to the indicator relating to the stage of development of the external quality assurance systems, 69% of the countries are in the green or the light green score category, representing fully functioning systems, applying to all higher education institutions, following the four-phase model (self-assessment, external review, publication of the results and follow-up procedures) in which the agency or agencies operating within the country had already undergone external assessment foreseen in the third part of the ESG (dark green) or had already set a date to do so (light green). Included in the yellow category are 29% of

30 Rauhvargers, A., Deane, C. & Pawels, W. (2009) . Bologna Process Stocktaking Report 2009.

22

countries, corresponding to cases in which the system does not cover all higher education institutions, or only covers two or three of the phases mentioned, or when no date has been fixed for the meta-evaluation of the Agency. There is only one case in the orange category, representing an ongoing situation at the very early stages of implementation of external quality assurance. In the indicators relating to ‘level of participation of students’ and ‘level of international participation’, 73% and 58 % of countries, respectively, are in the green category.

The ENQA survey illustrates that in spite of the predominance of the four-phase model, external quality assurance agencies have diversified organizational structures and operational methods, as can be seen from the following:

a) In terms of institutions assessed, 9% of the agencies only evaluate universities, 15% evaluate other higher education institutions, 42% evaluate both universities and other higher education institutions, 24% evaluate other types of institution, and 10% operate in a specific area of knowledge;

b) In terms of the geographical area of action, the range of operation of the majority of agencies is national; there are only seven countries within the European Higher Education Area where there is more than one Agency with responsibility for external quality assurance, which share tasks on a regional, procedural or type of target institution basis, as a rule with no overlapping of competencies (with the exception of the German and Dutch systems, and in part the Spanish system, where there is competition among the agencies operating within the country);

c) With regard to the processes used, there is a clear predominance of assessment and accreditation, followed at some distance by audit processes; almost two thirds of the agencies use assessment and/or accreditation processes at programme level and almost 40% at institutional level; the auditing processes are mainly used at an institutional level;

d) As regards the autonomy of the agencies to define the criteria and standards applicable to the processes of external quality assurance, 79% declared that they participate in the final decision concerning specification of these elements and in 62% of cases this decision is the Agency’s alone; in two cases only, this decision is taken by the government alone.

Closer study of the data collected in the survey reveals that almost all national agencies have more than one type of approach to external quality assurance and some use the three processes (assessment, accreditation and auditing). The survey report presents some interesting correlations31:

- “Agencies that conduct programme evaluation also conduct – at a rate of fifty percent – accreditation of programmes and evaluation of institutions.

31 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 24.

23

- Agencies that conduct accreditation of programmes also conduct – at a rate of fifty percent – evaluation of programmes and accreditation of institutions.

- Almost every Agency that accredits institutions also accredits programmes, and half of them also evaluate programmes and institutions, and audit institutions.

- Almost two-thirds of the agencies conducting audits at the institutional level also evaluate programmes, and half of them also accredit programmes, and evaluate and accredit institutions.”

The general impression gained from the survey is consequently one of well developed external systems of quality assurance and on the whole the agencies use different types of procedures, and have begun to add new activities to their portfolio, which reflects the dynamic nature of the evolution that has been seen in quality assurance in higher education. One particular facet of this evolution – which shows how external pressure, namely from political powers and public opinion, can influence QA systems – was the addition of programme accreditation procedures in systems which were previously oriented towards evaluation processes. This understanding is also expressed in the conclusion of the 2nd European Quality Assurance Forum, where it is stated that “with few exceptions such as Scotland and Finland, when quality assurance starts with an improvement focus, political pressure forces an accountability orientation into the system”32. A prime example of this effect was seen in Sweden in 2001, when after two cycles of institutional assessment, cyclical evaluation of programmes was introduced, largely as a result of pressure from student associations, who complained that institutional evaluation provided no information about the quality of the programmes33.

These changes of approach to external quality assurance, which have been translated into changes in the juridical framework in various countries, are among the most notable findings of the survey: 75% of agencies indicate that either they have recently changed their work approach or are about to do so, in the majority of cases in very significant ways. The reasons given for this are not only in order to respect the ESG but also the change to other procedures or, more often, the addition of another procedure. Generally speaking, a comparison with the 2003 report shows a broadening of the scope of action of the agencies. Some of these changes correspond to in-depth restructuring of agencies with a well-established tradition, as happened for example with the Danish Evaluation Institute (EVA), who saw its role changed as a result of the creation of a new accreditation organ, and the Comité National d’Evaluation (CNE), in France, which was 32 Harvey, L. (2008). Using the European Standards and Guidelines: Some Concluding Remarks. In EUA(2008), p. 84. 33 Serrano-Velarde, K. & Hopbach, A. (2007). European politics of quality assurance and the introduction of the Standards and Guidelines for Quality Assurance in the European Higher Education Area. In HRK (2007), p. 39.

24

incorporated in the new Agence d’Evaluation de la Recherche et de l’Enseignement Supérieur (AERES), which now covers not only all higher education institutions but also all research bodies.

In this brief overview of the dynamics of external quality assurance it is important to mention another point which has been under discussion since the Recommendation of the Council of Europe in 1998 and which concerns the standards applied to the agencies themselves, as organizations. Part 3 of the ESG was a response to this question, and had a strong impact on the organization of the agencies, as was seen previously. The importance of the standards established by the ESG as a frame of reference for the organization and operation of the agencies is also demonstrated by the founding of the EQAR, which uses these standards as a condition for registration.

The fact that the ESG were developed for use in the whole of the European Higher Education Area, regardless of structure, function and dimension of the different national systems, has nonetheless given rise to some problems, as is recognized in the report on the ENQA survey:

“Even though there is some convergence in European quality assurance, there are limitations when the requirements of ESG conflict with national regulations and traditions. (…) The analysis and commentaries [related to questions 11 to 24 of the survey, which paraphrase Part 3 of the ESG and thereby the ENQA membership criteria] acknowledge that there can be a strained relationship between national traditions, legislation and the ESG” 34.

These difficulties have led to the introduction of some adjustments concerning the application of the ESG, illustrated for example in the positions taken by the Board of the ENQA in 2006 with regard to the degree of compliance of the agencies with the ESG:

“Full or substantial compliance may be impossible for some agencies, owing to restrictions placed on them by the very nature of their work and/or legislation in place in their country (ies) of operation. When considering such cases, the ENQA Board will take mitigating circumstances such as these into account” 35.

Moreover, regarding the notion of independence of the agencies, which is one of the basic requirements of the ESG, the Board of the ENQA deliberated at the same time that ‘independence’ for the purpose of the ESG, should be understood as “autonomous responsibility for operations, where conclusions in reports cannot be influenced by HEIs, ministries or other stakeholders”36,37. 34 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 20. 35 ENQA (2006). Guidelines for national reviews of ENQA member agencies, p. 6. 36 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 33. 37 The evolution in ENQA’s position is interesting if we consider that not long before, in the external evaluation of the Portuguese evaluation system, one of the criticisms made by the evaluation team was

25

This change of attitude with regard to compliance with the ESG was corroborated in the London Communiqué, where as regards admission into the EQAR it was stated that this should be decided on a basis of substantial compliance with the ESG, which represents a move away from the original position of the ENQA, which foresaw compliance with all the standards as a condition for inclusion on the register38.

Another of the requirements imposed on agencies by the ESG was the compulsory establishment of procedures regarding their own transparency and accountability, including the existence of a duly publicized policy (with support documentation) for the effect, as well as the undertaking of an external assessment of the Agency at least once every five years. According to data collected in the survey, 89% of agencies stated they had already developed some type of internal procedures, at least partially. Among the measures adopted by the agencies, the following four emerge with a level of development between moderately effective and effective:

- The establishment and enforcement of a no-conflict-of-interest mechanism in the work of the agency’s external experts;

- Mechanisms for collection of feedback from experts and reviewed institutions, and for the analysis and reflection on such data, for the purpose of informing and supporting the agency’s improvement and development;

- Processes and results which reflect the agency’s mission and goals of quality assurance;

- An appeals system.

Regarding external assessment, in compliance with the ENQA membership criteria, only 14 agencies (out of 46) stated they had already been externally assessed, while 20 others indicated they were preparing their external assessment39. The 2009 scorecards presented in the Bologna Process Stocktaking Report in Louvain, also reflect the modest progress in relation to this item, insofar as only 16 countries had organized external assessment of their respective agencies, although a further 17 had set dates for it, and 15 had no plans in this respect, so the conclusion was drawn that “while the scheme of external QA has been widely implemented, in some countries it may not yet operate entirely in accordance with the ESG”40. More up-to date data reveal that 34 agencies have already undergone external evaluation, corresponding to only 17 countries. 41

the “apparently limited independence of the system”, when in reality, the concept of independence as expressed above was entirely respected. 38 In the words of Amaral and Rosa (2008, p. 79), “this lack of definition raises problems of clarity and comparability, which are very much in fashion today”. 39 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 80. 40 Rauhvargers, A. et al. (2009). Bologna Process Stocktaking Report 2009, p. 8. 41 Externally Reviewed Agencies, ENQA, http://www.enqa.eu/reviews_agencies.lasso, retrieved 14.05.2010.

26

The question of financing is another important point regarding European trends in the establishment of agencies, which will certainly be put to debate in Portugal.

As was mentioned in Chapter 2, external assessment derives from the social dimension of higher education and is seen by the political powers and by society as a crucial instrument with which to gauge the quality of institutions and study programmes. It is not, therefore, surprising that in the majority of countries the initiative for the establishment of external quality control measures came from governments or was supported by governments and in most cases, as would be expected, this is reflected in the funding of the agencies.

In reality, according to the ENQA survey, 78% of the agencies are mainly funded by the State (41% of these are exclusively funded by the government and in the case of 37%, the level of public funding ranges from 52-98%), and only 15% (6 agencies) do not receive any government funds (though it should be noted that four agencies included in the survey were international). Two agencies state that they are only supported by higher education institutions and a further two are exclusively funded by payment for services rendered, though the nature of these services is not specified. In 2003, for example, the Dutch agency NAO (currently NVAO) received public funding of 3.6 million Euros, and also charged the institutions 2,500 Euros for each process of programme accreditation42. So it may be concluded that the State is by far the main source of funding of agencies in the European Higher Education Area.

5.4 FROM QUALITY ASSURANCE TO QUALITY ENHANCEMENT

Quality assurance processes have four general aims –accountability, conformity, control and enhancement – which are all present in the external quality processes in Europe, although to different degrees and with frequent changes of emphasis in the various countries.

Accountability and continuous enhancement are aims which are included in the majority of national systems, but there are well-known difficulties in conciliating them. The ESG, which appear to be geared towards continuous enhancement, but which also admit other aims, do not resolve the latent incompatibility between the two approaches (accountability and continuous enhancement). One solution which seems to be gaining some popularity, although it is not an easy one to implement, is the trend to focus

42 Dittrich, K. (2004). Accreditation in the Netherlands. In Di Nauta et al. (2004), p. 56.

27

external quality assurance on accountability but to focus the internal processes on continuous quality enhancement43.

The report Trends V identified a number of serious problems in the balancing the need for the development of forward-looking, innovative internal systems of quality assurance and standardized accreditation procedures in use. This is mainly due to the difficulties caused by the formal accreditation procedures regarding innovation and curricular reform, interdisciplinarity and experimentation within the ambit of new courses adapted to the Bologna Process. In particular, in countries where the national accreditation systems function at programme level, there are often problems of uneasiness regarding emerging institutional procedures and strategies for quality enhancement44.

On this subject, in a paper presented at the opening of the 2nd European Forum on Quality Assurance, Sybille Reichert made the point that the development of quality in higher education is much more than “the formal quality assurance processes that policymakers like to focus upon when they speak about quality in higher education”. Quality enhancement brings together a wide range of institutional development methods and, on its positive side, “the Bologna reforms could improve quality in multiple ways: through the opportunities they offer to reflect and review curricula, to reform teaching methods (student-centred learning, continuous assessment, flexible learning paths) and even through strengthening horizontal communication and institutional transparency.”45

Improvement in the quality of teaching is, indeed, central to the Bologna Process and quality assurance processes must respond to the demands of society (at the risk of losing credit in the eyes of the public if they fail to do so), but must not be an obstacle to innovation and development. The dynamic evolution that the agencies have been registering, which has already been mentioned, would appear to imply that there is receptiveness to this concern, and cases have already arisen where there has been a movement away from accreditation of programmes46 or at least where this has been discussed, towards the adoption of approaches at an institutional level which are less intrusive in the detailed organisation of teaching.

43 “externally-addressed accountability through internally-organised enhancement”, in the words of Lee Harvey (2008). 44 Crosier, D. et al. (2007). Trends V: Universities Shaping the European Higher Education Area, p. 60. 45 Reichert, S. (2008). Looking Back – Looking Forward: Quality Assurance and the Bologna Process. In EUA (2008), p. 5. 46 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 85.

28

The changes occurring in the methods used by the agencies do not, however, only affect the nature of the procedures, but also more basic aspects, such as changing the emphasis from the input indicators to others which , in principle, have more to do with the learning outcomes, with the support of a wide range of objective criteria, which are interrelated, namely: (i) with the qualifications frameworks; (ii) with benchmarks related, for example, to Tuning; and (iii) with the ESG. Proper treatment of learning outcomes within quality assurance will continue to pose challenges to the agencies and in particular to higher education institutions. In effect, with increasing numbers of students from more varied social backgrounds, registering on more and more varied higher education programmes, there is obviously a great advantage in having more detailed information on the quality and the relevance of the teaching. The students themselves are more and more involved in the process of quality assurance and demand new sources of information. Employers also require better knowledge of what the graduates have learnt and what they can do. Consequently, emphasis on learning outcomes will tend to increase47.

In the concluding remarks of the ENQA survey, however, a question is raised: But can quality procedures cope with the amount of work required to provide reliable information on these at a detailed level? The conclusion drawn was “probably not if the emphasis on data collection is merely for accountability and reporting; but if quality assurance has quality improvement/enhancement as an agreed and truly integrated focus, then it is more likely to be able to”48.

The concern with finding a balance between quality assurance processes and innovation and creativity in teaching, in order to give more emphasis to quality enhancement, while maintaining the balance between different approaches to quality assurance, was the object of Project QAHECA49. This project has been developed over the last two years with the participation of about three dozen agencies and higher education institutions, with the aim of exploring how internal and external quality assurance systems can provide support for innovative and creative institutions, able to adopt and implement schemes for modernization of their teaching/learning processes.

In the final report on a previous EUA project, on the development of an internal culture of quality within universities (the Quality Culture Project, undertaken between 47 Ibidem, p. 88. 48 The report rounds this off with the following optimistic note: “it is interesting and encouraging to note that the survey shows, in several places and in different ways, such a change in emphasis from assurance to enhancement” (Ibidem). 49 This was the Quality Assurance for the Higher Education Change Agenda (QAHECA) Project, launched in 2007 by the EUA, in partnership with ACQUIN (Germany), the Higher Education Academy (United Kingdom) and the National University of Ireland, and recently completed. .

29

2002 and 2006), the participating institutions highlighted the risk of internal quality assurance processes becoming mere bureaucratic procedures, even when they are undertaken properly50. To study this issue in more depth and to attempt to better understand how creativity could be reinforced in higher education, the EUA launched the Creativity Project, during which it was concluded that:

“Quality processes have the potential to strengthen creativity and innovation if they are geared towards enhancement and focus on the capacity to change as a way to incorporate a future dimension. However, they can also have highly detrimental effects if they stress conformity over risk taking, are oriented towards the past rather than the future and develop into burdensome bureaucracies.” 51

These concerns were the starting point for QAHECA with a view to exploring the best ways to limit the potential negative effects of (internal and external) processes of quality assurance, and to steer these processes towards constituting an added value with regard to creativity in teaching and learning. One of the outcomes of the project was a set of non-prescriptive recommendations, putting forward some key principles to be considered by higher education institutions and by agencies in the planning of their current practices of quality assurance. Given the relevance of this theme in the current context of evolution of quality assurance in the European Higher Education Area, these recommendations are transcribed below:

Recommendations from the QAHECA project52 “1) First and foremost, quality assurance must be context sensitive and thus

individualised. When developing quality assurance processes HEIs and QA agencies need to take into account disciplinary characteristics, various organisational cultures, the historical position of the institution as well as the national context.

2) Quality assurance processes – both external and internal – should aim at enhancing the institutions’ capacity to change in order to reach the strategic goals of each institution better. Thus, we invite both QA agencies and HEIs to commit to a developmental approach in their quality assurance processes.

3) Quality assurance should be inclusive. A key success factor for an efficient QA that enhances creativity at institutional level implies engaging the whole institutional community and not just considering QA as the special purview of a specific QA unit. This approach regards, for example, strategic planning, educational development and staff development as part of QA processes. We also urge the QA agencies to revisit their standards and processes in order to analyse in which ways they can encourage institutions to adopt this approach.

4) Both HEIs and QA agencies should aim at ensuring the engagement and capacities of key actors in quality assurance processes. The role of the institutional leaders is to provide support and a framework for quality assurance

50 EUA (2006). Quality Culture in European Universities: A Bottom-Up Approach, p. 32. 51 EUA (2007b). Creativity in Higher Education, p. 8. 52 EUA (2009a). Improving Quality, Enhancing Creativity: Change Processes in European Higher Education Institutions, p. 20

30

and creativity. Through staff development, the staff of the institutions can be encouraged to assume an active role in order to ensure the implementation of the inclusive approach to quality assurance mentioned above, while at the agency level, the awareness and understanding of the staff on activities and developments at institutional level needs to be continuously promoted. And last but not at least, both HEIs and QA agencies need to foster greater student engagement through training and support in order for the students to be able to assume their role as key partners in quality assurance.

5) A precondition for an effective QA that enhances creativity is a partnership between institutions and agencies. This partnership will create space and trust for critical self-reflection which is a prerequisite for creating something new. Trust could be increased for example through confidentiality of institutional self-evaluation reports and developing external QA processes that are based on incentives rather than sanctions. We invite HEIs and QA agencies to work on building this partnership.

6) Quality assurance processes need to allow risk taking and failure which are essential for creating new knowledge. Internal quality assurance processes should be able to identify failures and define the process through which the institution reacts and rectifies the situation when a failure has taken place rather than prohibit risk taking altogether. For its part, external quality assurance should aim at checking if an HEI is capable of reacting to abnormal circumstances rather than sanctioning occasional failures.

7) Sharing experiences in QA is essential for the future development of quality assurance. We encourage the creation of platforms for both horizontal and vertical dialogue at various levels: within institution between departments, within a country between institutions, at European level between both HEIs and QA agencies, etc. While encouraging this dialogue, it should not be forgotten that when learning from others’ experiences, whether good or bad, one should never aim at merely copying successful practices, but at critically analysing which components of the practice might be applicable to one’s own context.

5.5 FUTURE DIRECTIONS AND CHALLENGES

The main factors that will continue to influence the mode of operation of the agencies are the ESG and also the EQAR, as well as the national qualifications frameworks. Apart from the crucial question of the implications of learning outcomes in quality assurance, other specific subjects are also on the agenda, such as. the development of methods of quality assurance for e-learning and joint-degrees and the mutual trust and recognition among agencies. The ENQA survey asked agencies about the foreseeable developments in their scope and modes of operation within a two-year period. Their replies point to a continuation of the same dynamic evolution which has been seen so far53:

- Over half of the agencies (27 out of 47) were planning to review their procedures, 22 with the introduction of different methodologies for external quality assurance;

53 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 79.

31

- Roughly half were awaiting developments in the national qualifications framework and approximately one third predicted alterations in the legislation governing higher education which would impact on the work of the Agency;

- Perhaps more significantly, almost a third of the agencies indicated that they were, or were going to be, involved in processes of reorganization or fusion with other agencies.

Thus, this panorama of continuous evolution does not only correspond to alterations to sporadic revision of procedures, although this does seem to be happening continuously, but rather to much more far-reaching changes, as was made clear in the previous point regarding emphasis on institutional development and quality enhancement. For example, the importance attributed by the Bologna Process to graduates’ skills and their employability would still not appear to be sufficiently reflected in the priorities of quality assurance procedures. It is likely that this question will take on more importance in quality processes as part of the shift of emphasis in the various quality assurance procedures mentioned previously.

One future direction which seems to be taking steady steps, as will be seen in more depth in the following chapter, is the possibility of a lighter touch approach in external assessment/accreditation at programme level, due to growing confidence in the quality assurance procedures developed within the higher education institutions. As is stated in the conclusions of the QAHECA project,

“Diversity of institutional missions and cultures will require more trust and flexibility from QA agencies. In this context, as HEIs develop a more mature and better-embedded quality culture, QA agencies will hopefully be able to focus on the effectiveness of institutional systems and adopt more developmental approaches.” 54

The Trends V report also points to the continuous development of assessment procedures and enhanced quality in higher education institutions as a considerable challenge for the future. The report considers that the trends are positive in this regard, insofar as institutions have begun to take on more responsibility for the quality of their educational offer. However, the report considers that there is still a lot to be done in this domain, and responsibility for this must be shouldered by institutions, governments, and agencies: institutions should not be complacent in questions of quality, which are essential for their competitiveness; governments should guarantee a legislative framework in which systems of quality assurance are not overly bureaucratic or excessively expensive and burdensome for the institutions; after a first cycle of external quality assurance procedures, agencies should tend to adopt a risk-based approach,

54 EUA (2009a). Improving Quality, Enhancing Creativity: Change Processes in European Higher Education Institutions, p. 20.

32

while remaining vigilant, bearing in mind that, as a rule, assessment regimes have concluded that the quality of higher education is generally satisfactory or even better. The authors of the report conclude that “The best guarantee of success is the efforts of autonomous and properly funded institutions that have well developed internal quality processes”55.

Although it is beyond the scope of the agencies, an emerging area which deserves attention is that of rankings, which has gained popularity among politicians and stakeholders. These rankings are, wrongly, taking on a function of quality assurance and they are seen as a measure of excellence by public opinion, influencing the behavior of higher education institutions in both positive and negative ways, particularly with regard to questions of course content, research priorities and organization56. The OCDE project AHELO, and the European Commission projects to classify European universities and to introduce a multidimensional ranking, should be followed closely, because of the potential effects of stratification of higher education institutions, with consequences also for the organization of quality assurance systems.

A final note in this brief outline of the challenges to be faced in the near future concerns the apparent inertia of the agencies (and also of institutions) regarding national legal frameworks and other factors which condition quality assurance. In the concluding remarks of the report Quality Convergence II, the authors make the following observation:

“Agencies are not used to questioning their own activities. They do not tend to challenge the framework that form the basis of their raison d’être and to question the political decisions that impact on their work. They resolutely see themselves as operators of a set of procedures and defend themselves against encroaching on the territory of those that they see as political decision makers or the academic world”57.

Thus the agencies would appear to be faced with the additional challenge of attempting to adopt a more pro-active stance before political decision-makers, in order to influence the development of more flexible quality assurance processes, even in circumstances in which the juridical framework in force makes this difficult.

55 Crosier, D. et al. (2007). Trends V: Universities Shaping the European Higher Education Area, p. 78. 56 Hazelkorn, E. (2009). The Emperor has no Clothes? Rankings and the shift from quality assurance to world-class excellence. In EUA (2009b), p. 18. 57 Crozier, F., Curvale, B. & Hénard, F. (2006). Final report on the pilot Quality Convergence II project: Promoting epistemological approaches to quality assurance. In Crozier et al. (2006), p. 27.

33

6. INTERNAL QUALITY ASSURANCE SYSTEMS – THE EUROPEAN PANORAMA

6.1 THE IMPLANTATION OF INTERNAL QUALITY ASSURANCE SYSTEMS

The central role of internal assessment in processes of quality assurance was highlighted in Chapter 2 of this work, according to the repeatedly emphasised principle that quality and quality assurance are primarily the responsibility of the higher education institutions themselves. European universities adhered unequivocally to this principle, as is expressed in the Declaration approved at the Graz Convention, which was a preparation on the part of higher education institutions for the Berlin ministerial meeting:

“The EUA proposes a coherent QA policy for Europe, based on the belief: that institutional autonomy creates and requires responsibility that universities are responsible for developing internal quality cultures and that progress at European level involving all stakeholders is a necessary next step. An internal quality culture and effective procedures foster vibrant intellectual and educational attainment. (…) With the active contribution of students, universities must monitor and evaluate all their activities, including study programmes and service departments. External quality assurance procedures should focus on checking through institutional audit that internal monitoring has been effectively done.” 58

This assumption of self-responsibility on the part of higher education institutions was echoed in the Berlin and Bergen Communiqués and particularly in the ESG, which not only strongly emphasized this principle, but have also contributed to its development in what concerns the following essential aspects, which are an important contribution for quality assurance in Europe59:

- While stressing that the primary responsibility for quality assurance lies with higher education institutions, the ESG add that external control should be lighter when internal processes reveal themselves to be robust, which is in line with a recommendation contained in Trends IV: “HEIs and QA agencies should cooperate in optimizing the relations and coordination between internal and external quality assurance processes, to alleviate the administrative burden on institutions without reducing the value for quality improvement. In particular, external quality assurance should be reduced in direct correlation to the evidence of robust internal quality processes”60;

- In accordance with the ESG, internal quality assurance is much more than the establishment of formalised processes, and should also emphasise a set of institutional and individual attitudes to foment a culture of quality and of continuous quality enhancement;

58 EUA (2003). Graz Declaration 2003 – Forward from Berlin: the Role of the Universities, p. 9. 59 Reichert, S. (2008). Looking Back – Looking Forward: Quality Assurance and the Bologna Process. In EUA (2008), p. 7. 60 Reichert, S. & Tauch, C. (2005). Trends IV: European Universities Implementing Bologna, p. 32.

34

- In consonance with the Bologna Process, the ESG reflect a change of emphasis from teaching in itself, as was traditional, to the interests of the students and other stakeholders, namely in regard to information and student support, the success of the graduates and the inclusion of students as active partners in the quality assurance (internal and external) processes.

With the transposition of the ESG to the national juridical regimes, which happened all over Europe, the duty of institutions to develop appropriate internal processes to deal with quality assurance became a formal obligation. However, according to the Bologna Process Stocktaking Report, presented in Louvain, progress in the implementation of internal quality assurance processes is slower than that seen with external processes. It was remarked that this could in part be because in some countries the internal dimension of quality assurance is still thought to refer only to the compiling of self-assessment reports for the purpose of external assessment61. It is emphasised that steady progress has been seen in relation to Part 1 of the ESG in areas which are well established in higher education institutions, such as internal approval of programmes and the publication of information. However, it was noted that the connection of the programmes to learning outcomes, as well as the development of assessment processes which measure how far the expected learning outcomes are achieved by students, are areas raising greater difficulties and which will take longer to implement. It is additionally stated that the national 2009 reports indicate that learning outcomes are not always understood as the specifications of what “the learner will know, understand and be able to demonstrate after completion of a programme of learning (or individual subject)”, and are often confused with the general aims of the programmes, which cannot be measured and consequently cannot be used in assessing students62.

According to the Trends V report, there is growing awareness on the part of higher education institutions of the potential benefits and challenges associated with activities undertaken in the area of quality assurance and quality enhancement, and there has been a good deal of investment in internal quality assessment systems, although there are few cases of institutions which are adopting a holistic approach to quality enhancement63. For example, the fact that only few countries developed assessment of student support services in the majority of their institutions demonstrates the need for concerted approaches to the enhancement of learning environments, for which these services are essential.

61Rauhvargers, A. et al. (2009). Bologna Process Stocktaking Report 2009, p. 8. 62 Ibidem, p. 8; 56. 63 Crosier, D. et al. (2007). Trends V: Universities Shaping the European Higher Education Area, p. 9.

35

The information which was gathered from the questionnaire which was the basis of the report clearly indicated that, in one way or another, some form of internal quality assurance activities are undertaken by a much higher proportion of institutions than seen four years previously in Trends III. Regarding the concrete questions dealt with in the questionnaire, the following results can be summarized64:

- Over 95% of the institutions which responded stated that they carried out internal assessment of their programmes and 72% do so regularly (although there may be some confusion in these data between systematic internal assessment and a mere self-assessment exercise as a preparation for external assessment, as was mentioned earlier);

- Assessment of learning support services (libraries, counselling or academic guidance services...) is much less frequent, and is only undertaken regularly by 43% of the institutions;

- Research activities are assessed on a regular basis in 65% of the institutions, but the research teams are only assessed in 48% of cases;

- When it comes to individual assessment of teaching staff, two thirds of the institutions have compulsory procedures for this effect, 17% have voluntary procedures, and 16% do not undertake this type of assessment.

Among mechanisms adopted by institutions, there is increasing use of surveys in the form of questionnaires filled in by students and graduates, and also the use of ever-more sophisticated information platforms to make available comparative internal data with regard to the performance of students and staff, based on criteria associated with efficacy and efficiency of teaching and learning, of research and of other activities. One significant finding of the report was that students are more and more part of the quality assurance processes within the institutions and that this is seen by many of them as a factor that promotes greater awareness of the problems of quality, as well as the involvement of all actors65. The 2009 scorecards show that students participate in quality assurance in a variety of ways, including internal assessment of teaching (in 37 countries), participation in the writing of self-assessment reports (31 countries) and by their voice being heard by external assessment teams during their visits (38 countries) 66.

The national reports for 2009 of the 48 countries participating in the Bologna Process show that in the majority of countries higher education institutions are working actively in the establishment of coherent internal quality assurance systems and that the following improvement has been made with regard to the application of Part 1 of the ESG67:

64 Ibidem, pp. 56-59. 65 Ibidem, p. 59. 66 Rauhvargers, A. et al. (2009). Bologna Process Stocktaking Report 2009, p. 62. 67 Ibidem, pp. 53-56.

36

- A total of 41 countries stated that either all of their higher education institutions (21 countries) or most of their institutions (20 countries) have systems installed for internal monitoring, assessment and periodic review of their programmes and degrees, and a variety of methods are in use;

- In a quarter of the countries all institutions describe their programmes in terms of learning outcomes and in almost one third of the countries this practice is undertaken by the majority of institutions, although some national reports state that it is important to give support to the institutions regarding better understanding and formulation of learning objectives, because in some cases these objectives are not yet in syntony with the Dublin descriptors or the Tuning agreement;

- The introduction of procedures for student assessment in accordance with the expected learning outcomes is, however, much less developed – in more than half of the countries, either it does not exist at all, or is practised in only some institutions. The comments which appear in some reports show that there is some confusion in relation to this question, for example, when the assessment of learning outcomes is identified with a simple national scale with criteria defined for each of the values on the scale, which are indiscriminately applicable to all programmes or scientific areas;

- With regard to publication of up-to-date, impartial and objective information concerning programmes and degrees awarded, 30 countries state that this is practiced in all institutions and 16 countries state that this procedure is adopted in the majority of institutions.

With the objective of promoting the alignment of quality assurance systems with the ESG, in more than half of the countries stakeholders were consulted, following which the majority of these countries (25 in all) have introduced financial incentives for institutions for the improvement of internal quality assurance processes, in some cases with substantial amounts of money (it is mentioned that in one case this amount was over 30 million Euros68), namely for the creation or reinforcement of internal quality units or the audit of internal quality systems.

6.2 THE VALORISATION OF INTERNAL QUALITY ASSURANCE SYSTEMS

The ENQA report on the ESG emphasises the importance of internal systems of quality assurance, namely when it includes, among the general principles of good practice in processes of external quality assurance, the remark that “use should be made, wherever possible, of the results of institutions’ own internal quality assurance activities”69

Thus, as was mentioned in Chapter 3, the first standard of part 2 of the ESG establishes that external evaluation processes should take into consideration the efficacy 68Ibidem, p. 51. 69ENQA(2005). Standards and Guidelines for Quality Assurance in the European Higher Education Area, p. 15.

37

of the internal quality assurance processes, which puts additional pressure on the institutions for the development of their internal quality processes. The ENQA survey, published in 2008, includes an analysis of different areas which are subject to observation by the external quality assurance procedures, both at institutional and programme level, and identifies in both cases the five areas considered most important by the agencies. The conclusions were as follows70:

- With regard to assessment of programmes, 95% of the agencies assess the internal quality assurance procedures and this item appears in fourth place in the order of importance attributed to it, but with an impact factor which is close to the three preceding items (qualification of teaching staff, curricula, and installations and resources) and quite far ahead of many other items, namely those related to organisation of learning and employability;

- In external processes at institutional level, internal quality assurance procedures are assessed by 92% of the agencies and this is considered to be the most important item.

These data reveal that the work of the agencies is closely aligned with the above-mentioned standards. This is the case both regarding explicit appraisal of internal quality assurance systems during external assessment procedures, and also regarding the importance given to this assessment. However, the the centrality of internal quality assurance processes has a much broader scope, as was seen in the previous point regarding the expectations of higher education institutions and other agents, namely as expressed in the EUA statements and in the Trends reports, regarding the idea that the robustness and efficacy of the internal quality assurance processes, when duly confirmed, should lead to a climate of mutual trust between the agencies and the institutions, and consequently, through a risk-taking approach, allow progressive establishment of lighter and less intrusive external procedures. But are the agencies taking concrete steps in this direction?

This question is closely linked to that of the evolution of the external quality assurance procedures towards a greater emphasis on continuous enhancement, which was dealt with in 5.4, as well as to concerns voiced regarding the potential perverse effects of quality processes on the development of the Bologna Process. In the Trends V report, the section dedicated to quality assurance finishes with a quite emphatic statement on the subject:

“Key issue: Many higher education institution systems are currently being hold back from Bologna implementation – and thus from offering improved services to students and society – by national QA systems that are costly, offer no evidence of

70 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, pp. 47-50.

38

overall quality improvement, and stifle institutions’ capacity to respond creatively to the demands of evolving European knowledge society.” 71

Another relevant issue, highlighted in the reference above, relates to the cost-benefit analysis, including in the costs not only the direct financial costs of quality assurance processes but also the effort involved. Although not enough impact studies are available, it is beginning to be felt that the repetition of cycles of programme quality assessment or accreditation, using identical methodologies, is of little benefit to quality enhancement, and does not justify the costs and effort involved72. On the other hand, in the case of internal quality assurance processes, there is the perception that monitoring and assessment processes, without any effective link to the curriculum development processes, are not enough to guarantee the quality of teaching73, and thus there is a demand for more holistic approaches geared towards quality enhancement.

The ENQA survey provides some indications regarding the previously mentioned question of whether the agencies are taking these concerns into consideration. As part of the profound evolution that has been taking place in the scope of action of the agencies, which was mentioned in the previous chapter, some changes are occurring in the division of responsibilities between the agencies and the institutions. There is a trend for the institutions themselves to take responsibility for assessment of courses, whereas the quality assurance of programmes is undertaken, according to each case, either by the institutions or the agencies. The quality assurance procedures at institutional level are, as a rule, the responsibility of the agencies. These winds of change appear to have brought a shift in the balance towards greater effective responsibility on the part of higher education institutions, “with shifts away from external programme evaluation either having taken place or taking place, or being discussed; although there are exceptions”74 as is highlighted in the report on the survey.

This change of paradigm towards more flexible and more differentiated quality assurance systems, in which the internal mechanisms of quality assurance are made more robust and reliable, with the consequent lightening of external procedures, poses a great challenge for constructive cooperation between the agencies and higher education institutions, with responsibilities on both parts: the agencies must seek to ensure that the procedures set in motion by them support and help the development of initiatives of the

71 Crosier, D. et al. (2007). Trends V: Universities Shaping the European Higher Education Area, p. 61. 72 Klaassen, L. (2009). NVAO: Introducing, developing and implementing a new phase in the accreditation system in the Netherlands and Flanders. In EUA (2009a), p. 38. 73 EUA (2009a). Improving Quality, Enhancing Creativity: Change Processes in European Higher Education Institutions, p. 13. 74 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 86.

39

institutions without imposing a heavy bureaucratic burden which is foreign to the culture of the institution; the institutions, in turn, should seriously consider the establishment of a culture of quality which is embedded in the life of the institution, and which is recognised by the stakeholders; both should commit themselves to the construction of a climate of mutual confidence, which is crucial for the adoption of risk-taking approaches.

In this regard, it is worth listing some of the requirements for a culture of quality, to be understood as an iterative process which is at the same time a prerequisite for quality assurance and also an outcome of quality assurance. Key factors in quality culture are75:

“- academic ownership and engagement; - recognition of the need for a quality system (but not one driven by

bureaucracy); - a focus on changing people’s behaviour rather than the mechanics of a system

of reporting and review; - clarity of purpose; - centrality of students: both the student learning experience as the focus of

quality of education and the involvement of students in evaluative processes as credible evidence providers and peers in evaluation teams;

- encouragement of partnership and co-operation; - focusing less on individual performance and enabling community engagement

and team working; - a leadership style that inspires rather than dictates; - welcoming of external critical evaluation; - an integrated and continuous process of self-reflection; - providing the context to take the initiative to improve, even where it is risky”.

The aim of these characteristics is to promote a culture of quality which is at the same time both responsive, i.e., able to promptly respond to external pressures combining transparency and improvement, and regenerative in the sense of being geared towards the institution, well embedded in the internal community and accepting experimentation and risk-taking.

Internal quality assurance systems should naturally be based on these same principles, which presuppose a bottom-up approach, rooted in the awareness and involvement of academic communities, in the understanding that quality assurance activities are not activities which run parallel to the life of the institution, as the responsibility of only one or several specific people, but rather the responsibility of each and every member of the community, in a concern with quality which should be present in all the activities of the institution. Thus, one could consider as success factors for an internal quality assurance system, as identified by the EUA Quality Culture project76: 75 Harvey, L. (2008). Using the European Standards and Guidelines: Some Concluding Remarks. In EUA(2008), p. 84. 76 EUA (2006). Quality Culture in Universities: A Bottom-Up Approach.

40

appropriate strategic planning, the creation of organizational structures appropriate for quality assurance, the commitment of institutional leaders at the highest level, the involvement of staff and students of the institution, the involvement of external stakeholders, and a well-structured system of data collection and analysis.

It should be noted that a key factor in quality, associated to the role of leaders, is the importance of establishing clear aims for the design and implementation of an internal quality system. A recent case study on a project undertaken by the University of Leeds, United Kingdom, with the aim of promoting enhanced quality of teaching, concluded that the clear definition of a ‘sense of direction’, when properly supported by institution’s leadership, has an encouraging effect, promoting the following vital elements for enhancement to take place: 77

“- the building of trust across the institution such that quality assurance can effectively lead to enhancement and not be seen as bureaucracy;

- the development of a culture of time and opportunity which gives staff the time, scope and confidence to try new things defined by themselves and not by external drivers;

- the seeking of ways to empower academics by giving back to them the responsibility for the quality of their teaching;

- the development of a community where learning and teaching are as exciting and rewarding as research.”

6.3 AUDITING INTERNAL QUALITY ASSURANCE SYSTEMS

As was mentioned in Chapter 5, the 2008 ENQA survey identified the endorsement or discussion, in a number of agencies, of a move away from programme accreditation, towards more institutional approaches focused on auditing of internal quality processes of the institutions. It is, however, stressed that the dominant external quality assurance processes are still assessment and accreditation (of institutions and/or programmes), and it was noticed that auditing mechanisms – understood as procedures aiming at the assessment of strong and weak points of quality assurance mechanisms within the institutions – are the dominant method in fewer agencies78.

The Trends V report, taking for granted that the relation between institutional responsibility, transparency and autonomy, on one hand, and the need for trustworthy and transparent quality assurance mechanisms, on the other, seems to be already firmly established and understood, also concluded that there seems to be a trend towards external quality assurance mechanisms which are becoming less intrusive as the

77 Hodgson, K. (2009). Developments to encourage quality enhancement: a case study. In EUA (2009b), p. 23. 78 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 83

41

institutions themselves become more responsible and transparent, adding that: “This is reflected by the number of mature quality assurance systems which have moved away from a system of programme accreditation, replacing it with a focus on institutional evaluation or audit” 79. This would appear to be an iterative process, to the extent that when an Agency decides to adopt institutional audit procedures, the institutions begin to pay much more attention to their internal quality assurance procedures, in preparation for the external audit. Thus, a synergy is created between audit processes and internal quality mechanisms, which is reinforced in the case of agencies which combine the functions of consultancy and support to the institutions with the formal auditing process, although these are, of course, independent procedures.

The advantages of such an approach are clear, as was pointed out in previous points, both in economy of time and resources and in the potential advantages arising from greater emphasis on the promotion of quality, to the detriment of more formal analysis of conformity. At the same time, this is in line with the expectations regarding the self-responsibility of higher education institutions, and is more in accordance with the principle of institutional autonomy. However, such a fundamental transformation in external quality assurance mechanisms is only possible if there first exists a climate of confidence in the quality of the teaching, both on the part of the authorities and of public opinion, which will, in principle, demand at least one cycle of assessment/accreditation with real consequences, duly understood by society.

This issue has been the subject of discussion and of increasing attention on the part of agencies, which are increasingly including assessment of internal quality assurance systems among their external assessment procedures. This comes as the natural result of the fact that – and this is true in almost all European countries –institutions are bound to establish procedures geared towards quality assurance and quality enhancement. Some agencies have already evolved towards models of external assessment less centred on assessment of individual programmes, in connection with the development of institutional audit processes, which increase confidence in the efficacy of the internal quality assurance processes of the institutions, though with approaches and effects which are quite diversified. A group of agencies with more experience in this area have been meeting periodically to exchange ideas and has become an informal network of agencies that promote models of institutional audit80. 79 Crosier, D. et al. (2007). Trends V: Universities Shaping the European Higher Education Area, p. 61. 80 This is the Quality Audit Network, involving nine Agencies which met in Vienna, in April, 2008, and again in Madrid, in April, 2009, to consider in more depth specific aspects related to audits, sharing knowledge and experiences. The countries and Agencies involved are: Germany (GAC), Austria (AQA), Denmark (EVA), Spain (ANECA and AQU Catalonia), Finland (FINHEEC), England, Northern Ireland and Scotland (QAA), Norway (NOKUT) and Switzerland (OAQ).

42

In the following chapter an analysis is presented of some of the cases identified as good practice in institutional auditing, in the hope that this will be useful in the Portuguese context, both for specification of key aspects related to internal systems of quality assurance and for the development of a model for institutional audit.

43

7. INSTITUTIONAL AUDIT PROCESSES IN EUROPE

7.1 INSTITUTIONAL AUDIT PROCESSES – CASE STUDIES

A number of countries have developed processes of institutional auditing with a variety of approaches and aims, and which are at different stages of development – some agencies already have several years of experience of quality auditing, while others have only recently begun to develop these processes. In some countries, institutional auditing is the central element in quality assurance processes, while in others these audits are only one component of a larger set of external quality assurance processes. Although there are a number of common trends, which have been dealt with in previous chapters, the situation is so varied that an understanding and comparison of the different models of quality audit is only possible by looking at the context of the evolution of higher education policies and of quality assurance in each country.

In the sub-sections below there is a brief presentation of various case studies which reflect the dynamics of the evolution of external quality assurance in Europe, relating to national systems in which the focus is on institutional auditing. An important consideration here is an understanding of the context in which these processes were developed, their key components and any difficulties encountered, as well the extent of their contribution to simplification of external quality assurance procedures. The first to be presented are those in which the auditing systems are more consolidated as they have been operating for a number of years (as is the case of England/Northern Ireland, Scotland and the Scandinavian countries). Consideration is also given, however, to systems where auditing was introduced more recently or is still in the process of being set up (as is the case in Spain).

7.1.1 ‘Academic Infrastructure’ – External standards for quality assurance in the United Kingdom

The higher education system in the United Kingdom allows universities – which are not even considered as belonging to the public sector – a great deal of autonomy. Consequently, the government has no direct capacity of intervention in matters concerning higher education institutions, but an indirect link exists through the funding councils, which stipulate an external framework of reference for strategic action within the institutions81.

81 Serrano-Velarde, K. & Hopbach, A. (2007). European politics of quality assurance and the introduction of the Standards and Guidelines for Quality Assurance in the European Higher Education Area. In HRK (2007), p. 41.

44

The United Kingdom was among the pioneers in quality assurance in higher education, which gained ground at the beginning of the 90’s, and was very much concerned with accountability before government and society. In 1992, the Further and Higher Education Act restructured the organs responsible for funding, and created the new funding councils for England and Northern Ireland, for Scotland and for Wales, which became also responsible for quality assurance in teaching and learning with a very close association between assessment and funding. This situation lasted for four years.

In 1997 the Quality Assurance Agency for Higher Education (QAA) was established, with the statute of an independent body which, on a contractual basis, took over the functions of the funding councils in the area of quality assurance, promoting external review of degree programmes in all higher education institutions and safeguarding the public interest through the establishment of sound standards for higher education qualifications. This initiative of attributing to the new Agency a function which is essentially that of external control arose in the context of increasing public interest in the quality of degrees and qualifications awarded by the institutions.

The definition of methods and criteria for quality assurance processes is the direct responsibility of the QAA. The Agency began by adopting two complementary strategies for quality assurance, institutional review and teaching quality assessment82. However, the institutions enjoyed enormous autonomy, namely the ability to self-accredit their programmes and reacted badly to the external assessment exercise. There was widespread criticism of the QAA among the academic communities for obliging them to waste time and resources on what they considered to be a bureaucratic process of conformity which required a time-consuming process of data collection, and for interfering in their academic autonomy, with potential negative consequences in the processes of continuous development and enhancement83. This type of concern found some echo among official bodies and led to the creation, in 2002, by the Higher Education Funding Council – England (HEFCE), of a committee responsible for promoting a review of the measures supporting quality enhancement in teaching and learning84. The QAA accompanied this exercise and at the end of the first cycle of assessment of programmes, which ended in 2001, TQA was abandoned and the QAA began to use lighter methods related to external assessment at institutional level. 82 The “Teaching Quality Assessment” (TQA) was a programme conceived for the evaluation of all programmes in the United Kingdom. 83In this regard, the position of the Russell group of universities, which includes prestigious institutions such as the Universities of Oxford, Cambridge and Imperial College, gained prominence as they refused to continue with the process underway at the time. 84 The “Teaching Quality Enhancement Committee”, which published an intermediate report in 2002

45

However, this included some sampling auditing of scientific areas, which did not put an end to criticism of the system85. Thus, in 2006 a new auditing scheme was introduced, explicitly geared towards an institutional strategy for enhancement. In its handbook for institutional audits, the QAA stresses that its aim is “to support and not undermine institutions in their work to enhance the learning opportunities they make available for students” and defines ‘enhancement’ as “the process of taking deliberate steps to bring about improvement in the effectiveness of the learning experiences of students”.86

This system of quality assurance in the United Kingdom placed the main responsibility for academic standards and quality of teaching in the hands of each of the higher education institutions, which are independent and self-governed. This does not mean, however, that there are no strong external references for internal quality assurance. Indeed, the evolution of the QAA processes towards ones which are less intrusive at programme level was accompanied by the development – in close collaboration with the institutions and other stakeholders – of a broad external framework of reference for quality assurance within the institutions, known as the Academic Infrastructure. This is a set of guidelines and reference points agreed upon by the institutions, with four components, three of which have to do with the establishment of standards and the other with quality management. They are:

- Frameworks for higher education qualifications, which describe the levels of achievement and corresponding characteristics of the main qualification awards (for example bachelor with honours, or master) in a way which is comparable with the European Framework of Qualifications;

- Subject benchmark statements, which indicate degree standards to be expected in a range of subject areas, i.e., they describe the elements which provide an area of knowledge with its coherence and identity and define what could be expected of a graduate, in terms of competences and skills necessary to develop knowledge or competence in this area;

- Programme specifications, which contain concise indications concerning the intended learning outcomes in a given study programme;

- The code of practice for the assurance of academic quality and standards in higher education, which contains guidelines for good practices in the management of academic standards and quality, indicating key issues to be considered by the institutions in the following aspects, dealt with individually in each section of the code: postgraduate research programmes; collaborative provision and flexible and distributed learning (including e-learning); students with disabilities; external examining; academic appeals and student complaints on academic matters; assessment of students; programme design, approval,

85 Hodgson, K. (2009). Developments to Encourage Quality Enhancement: A Case Study. In EUA (2009b), p. 19. 86 QAA (2008). Enhancement-led Institutional Review Handbook: Scotland (2nd edition), p. 10.

46

monitoring and review; career education, information and guidance; work-based and placement learning; admissions to higher education.

Another type of external academic control is added to these four elements of reference for the organization of teaching and of quality, which is the long-established practice in the United Kingdom of having external examiners involved in student assessment, as a matter of principle.

Currently, the QAA undertakes the following procedures for external quality assurance in universities, which take into account the specificities of the organization of higher education in each of the four countries which make up the United Kingdom (there are specific procedures for Further Education Colleges):

- Institutional audit, for the universities of England and Northern Ireland; - Institutional review, in Wales; - Enhancement-led institutional review, in Scotland; - Collaborative-provision audit, for institutions in England and Northern Ireland

which offer a large number of study programmes through other institutions.

Students and learning opportunities are the main focus of institutional audits. Aspects of the handbook for audits adopted in 2006 were revised in September, 2009, in areas concerned with the collaborative offer of programmes and the participation of students in quality assurance processes. According to the new version, audit teams in England and Northern Ireland will now include students, which was not the case in the past.

Institutional audits have the concrete aim of examining the following aspects87: - “The effectiveness of an institution’s internal quality assurance structures

and mechanisms, in the light of the UK Academic Infrastructure and the Standards and Guidelines for Quality Assurance in the European Higher Education Area (…) and the way in which the quality of its educational provision and academic standards of its awards are regularly reviewed and resulting recommendations implemented. This provides public information on an institutions’ soundness as a provider of HE qualifications of national and international standing.

- The effectiveness of arrangements for maintaining appropriate academic standards and enhancing the quality of postgraduate research programmes.

- The effectiveness of an institution’s approach to building systematically upon the outcomes of their quality assurance procedures, on the findings of reports of external reviews, and on other information and feedback from students, graduates and employers, in order to develop and implement institutional approaches to enhancing the quality of provision.

- The accuracy and completeness of the information that an institution publishes about the academic standards of its awards and the quality of

87 QAA (2009). Handbook for Institutional Audit: England and Northern Ireland, pp. 2-3.

47

its educational provision, including the published TQI [teaching quality indicators]”.

It should be noted that the appraisal expressed by the audit teams does not directly concern academic standards themselves, or the quality of learning opportunities88, but rather the methods used by the institution to guarantee that good practices are assured in relation to these two core areas, as regards the reference points mentioned above, which are central to the quality system. The results of the audit are expressed as a qualitative appraisal of ‘confidence’, ‘limited confidence’ and ‘no confidence’ in each of the two aspects mentioned (quality assurance of standards and of learning opportunities) according to objective criteria and evidence-based justification, which are presented in detail in the handbook for audits89.

The audit reports also include recommendations for consideration by the institutions and explicitly identify features of good practice which the team considers to make a particularly positive contribution to the institution’s approach to the management of the security of academic standards and of the quality of provision, in the context of that particular institution. These recommendations are categorised, in order of priority, as:

- ‘Essential recommendations’, regarding important matters which are compromising quality or standards, and which consequently require urgent corrective action;

- ‘Advisable recommendations’, referring to issues which could potentially compromise quality or standards, and which require preventive or corrective action;

- ‘Desirable recommendations’, referring to issues which the team considers could contribute to enhance quality.

The QAA audit process does not lead to a formal decision of accreditation but, nonetheless, has consequences which are expressed in the nature of the recommendations made and in the follow-up measures:

- In the case of an appraisal of confidence, there can be no essential recommendations, though there may be some (few) advisable recommendations and possibly some desirable recommendations, and there will be a follow-up in the middle of the cycle, usually three years after the audit, in the form of an analysis undertaken by two senior members of the Agency, based on a concise report prepared by the institution.

- An appraisal of limited confidence is usually accompanied by some recommendations considered essential or by a large number of advisable recommendations, along with other recommendations. The institution is then requested to present to the Agency an action plan for implementation of the

88 In the context of the Academic Infrastructure, the expression learning opportunities is used to denote the combined effect of the programmes offered, and the academic and personal support given to students (Ibidem, p. 6.). 89 Ibidem, pp. 48-50.

48

recommendations, within three months following the publication of the report, and to later submit a progress report. The audit process is not deemed to be complete until the Agency considers that the plan has been executed successfully. If, after eighteen months, there are still doubts about the efficacy of the corrective measures, a new visit is made to the institution. There is always a follow-up report in the middle of the period.

- A no confidence result is always accompanied by essential recommendations, among others, and the institution is also requested to present an action plan for implementation of the recommendations, within three months, to be implemented within eighteen months, with submission of three-monthly progress reports. After eighteen months, there is a follow-up visit and if doubts remain about the efficacy of the plan, the QAA will anticipate the next audit. If the audit process terminates successfully, there will be a follow-up visit in the middle of the period.

In brief, there are two particularly noteworthy aspects of the evolution of quality assurance policies and processes in the United Kingdom, when compared with the wave of tighter control over programmes that has been seen in the majority of Western European countries throughout the last decade.

The first is the fact that, in a system which began by being one of the most geared towards accountability, no formal accreditation mechanisms have been introduced. The official position on this matter is made clear in the national report for 2005 for the Bologna Process, where it is stated that the institutional audit approach provides public information on teaching which if significantly better than that transmitted by a mere stamp of accreditation90. The responsibility for quality control of teaching and learning is unequivocally attributed directly to the institutions.

On the other hand, although British universities enjoy enormous independence, the principle of autonomy does not conflict with the existence of external rules and influences on the internal quality assurance systems. On the contrary, the specificity of the British quality assurance system lies in its strong external component, expressed in the ‘academic infrastructure’, to some extent of self-regulatory nature, which is incorporated within internal quality mechanisms91.

7.1.2 Quality Enhancement Framework’ – A radical approach to quality in Scotland

Within the context of devolution of competences effected by the British government, the model adopted by Scotland to manage quality assurance in higher education followed a different path from that of England and Northern Ireland, described above. 90 Green, R. (2005). National Reports 2004-2005. England, North Ireland and Wales, p. 5. 91 Serrano-Velarde, K. & Hopbach, A. (2007). European politics of quality assurance and the introduction of the Standards and Guidelines for Quality Assurance in the European Higher Education Area. In HRK (2007), p. 43.

49

The systems share some common reference points, namely regarding the Academic Infrastructure and the use of external examiners, and also in the way in which the appraisal is expressed by the audit teams regarding the level of confidence in the degree of efficacy of the internal quality assurance system (‘confidence’, ‘limited confidence’ and ‘no confidence’), thus ensuring the comparability of results of the different audit systems in use in the United Kingdom.

In 2003, after more than a decade of experiments with audit and assessment processes, a radical approach to quality assurance and enhancement was introduced in Scotland – the Quality Enhancement Framework (QEF) –, based on the idea that ‘the purpose of quality systems in higher education is to improve student experiences and, consequently, their learning’92.

The QEF consists of five interrelated elements93: - “A comprehensive programme of subject reviews, to be run by

institutions themselves; - Enhancement-led Institutional-level Review [ELIR], which will involve

all HE institutions over a four-year cycle; - Improved forms of public information about quality, based on addressing

the different needs of a range of stakeholders, including students and employers;

- A greater voice for student representatives in institutional quality systems, supported by a new national development service94;

- A national programme of quality enhancement themes, aimed at developing and sharing good practice in learning and teaching in higher education.”

The most innovative aspect of the QEF was the introduction of national themes, which are developed according to annual plans defined by the Scottish Higher Education Enhancement Committee, where the academic pro-rectors of the different universities have a seat, as well as students, and which has the technical support of members of the QAA Scotland. The aim of this initiative is the shared construction of knowledge and experience, which benefits from practices, evidence and experience, which are both national and international, constituting a valuable resource of benchmarks for institutions and for students in the development of processes of quality management95.

92 QAA (2008). Enhancement-led Institutional Review Handbook: Scotland (2nd edition), p. 1. 93 Davidson, A. (2009). Future Directions for the Scottish Enhancement-led Approach to Quality. In EUA (2009b), p. 31. 94 Students Participation in Quality Scotland (sparcs). 95 To date, six themes have been dealt with and they are: assessment and integrative assessment; responding to students needs; employability; flexible delivery; first year: engagement and empowerment; research-teaching linkages: enhancing graduate attributes. Recently a new theme was launched –

50

The system of institutional review – ELIR – focuses on the strategic management and efficacy of its implementation with a view to ensuring academic standards and to enhance the student learning experience, with an underlying objective to support the capacity for self-assessment and reflection within the institution. The assessment exercise covers all the credit-bearing provision within the institution, regardless of whether it is or not part of a degree programmes. The ELIR method consists of four integrated elements:

- An annual discussion with each institution, with the participation of a member of the Agency and a small group of senior members of the institution, to discuss approaches to quality developed throughout the year and the lessons being learnt from them, according to the corresponding results, as well as other relevant developments regarding quality enhancement that the institution may wish to introduce into the discussion (the annual meeting does not imply the preparation of any specific report by the institution, nor does it result in the expression of any formal appraisal or public report);

- The submission of a self assessment report (reflective analysis) every four years for effects of external assessment, in which the analysis is evidence-based;

- An external review based on the self-assessment report and on two visits by the audit team (the first lasting two days, and the second between three and five days), followed by the preparation of a report which, after validation, is published in two formats – the complete report and a summary;

- The dissemination of the results of the ELIR process and of good practices identified on the basis of the assessment undertaken.

A year after the publication of the report, the institution produces a follow-up document, indicating the measures taken as a consequence of the assessment and the respective efficacy. This document is discussed at the following annual meeting.

Both the complete reports and the summaries are presented to the Scottish Funding Council (SFC). If the results express limited confidence or no confidence, the SFC requests the institution to undertake remedial action in specific areas and a schedule and methods will be agreed upon between the SFC, the QAA Scotland and the institution for a follow-up review, which will make it possible to rectify the problems detected.

The quality assurance model adopted by Scotland is, therefore, very dynamic, focusing on specific quality enhancement themes which are widely discussed and participated, encouraging the active involvement of all the actors, a dialogue between institutions and stakeholders and the diffusion of good practices. What differentiates the QEF, and implicitly the ELIR, are the effective partnership established between the

Graduates for the 21st Century: integrating the Enhancement Themes – to integrate the results of the themes dealt with previously (Carney & Macpherson, 2009).

51

QAA Scotland and the SFC, theRectors’ Conference (Universities Scotland), each one of the institutions and the student organizations, as well as the fact that emphasis is given to consensual approaches, rather than coercive ones which are embedded in various external quality assurance systems. As is stated in the final report on the cycle of external assessments of 200796, the QEF represents a strong commitment to a culture shift in a move towards continuous quality enhancement97.

7.1.3 ‘Quality Reform’– The Norwegian system98

There is a strong predominance of the public sector in the higher education system in Norway, which covers over 90% of the student population in higher education. The private sector is made up of small institutions, mostly operating within specialised areas, and predominantly offering almost exclusively 1st cycle programmes. All institutions, both public and private, meet common academic standards and criteria.

The juridical framework of higher education in Norway was revised in 2002, at the time of the introduction of the Bologna reforms, which were integrated into a broader process of ‘quality reform’ of higher education99. The main alterations introduced by this reform concerned the adaptation of degree structures to the Bologna Process, the strengthening of institutional autonomy, the imposition of the obligation on the part of institutions to provide students with active support, and the introduction of a formal system of accreditation for all institutions, and more specific requirements in the area of quality assurance.

The Norwegian Agency for Quality Assurance in Education (NOKUT) was then created and began its work in January 2003, as an independent Agency responsible for all those decisions concerning accreditation outside the self-accreditation powers conferred on the higher education institutions themselves.

96 Enhancing the quality of teaching and learning in Scottish Universities 2007. Final external evaluation report quoted in SFC circular SFC/11/2007 ‘Evaluation of the higher education quality enhancement framework: final report’. 97 This culture shift is described in the report thus: “A culture shift away from top-down compliance-inducing processes to participative and critical supported self-evaluation; away from audit and towards improvement; away from ruffling the surface of higher education practices and towards permeating the system with practices compatible with the QEF; away from mechanistic models based solely on inputs and outcomes and towards more sensitive other forms of evidence of cultural change, while maintaining rigour and challenge”. 98 This section has been updated in early April 2011, based on a direct contact with Wenche Froestad, at NOKUT. 99 Holmen, T.B. (2004). Quality Assurance of Norwegian Higher Education. In Di Nauta et al. (2004), pp. 59-61.

52

Accreditation became compulsory for all institutions, both at institutional and programme level. Accreditation decisions do not have limited validity, and remain valid indefinitely unless specifically revoked following a negative appraisal.

Institutional accreditation confers on the institutions the right to award national degrees or diplomas in the study cycles corresponding to the respective categories (all cycles, in universities and in specialized universities and only the 1st cycle in university colleges). The new legislation considered that all public institutions were automatically accredited. The private institutions, however, had to undergo a process of accreditation by NOKUT, which also covers the creation of new institutions and also the upgrading of statute requested by institutions which are already accredited.

An institution may function without being accredited, as long as its study programmes undergo an individual process of accreditation. University colleges may also offer programmes above first-cycle level if these programmes are accredited individually. In the case of accredited institutions, all programmes (both those already operating and new programmes) are automatically considered to be accredited and the institutions themselves are held responsible for the quality of their provision, i.e., for the self-accreditation of their programmes.

On the other hand, accredited institutions became subject to a cyclical process of institutional audit, no less than every six years, to ascertain that the internal systems of quality assurance do effectively include mechanisms capable of ensuring that their educational activities are of a high level, that they are carried out with a view to continuous enhancement, that they are able to reveal cases of inferior quality, and to discriminate between good and bad quality, to provide a basis for self-assessment and institutional development, and to help to promote a strong quality culture. If audit leads to a negative result, the institution will not be able to create new programmes, but does not immediately lose accreditation – it is given a period of six months to correct the situations which were judged to be lacking. Following a new revision of the juridical framework of higher education, these demands have now to be met by all providers of higher education in Norway.

Another mechanism for the control of accreditation is the process of revision of accreditation, which promotes the evaluation of a programme, a subject area or even the institution as a whole, as a consequence of the results of audits or other relevant evidence, and also through spot checks. A negative result in this revision results in loss of accreditation. From 2011 onwards (following another revision of the juridical framework), NOKUT will also use general supervision in the quality assurance of

53

ongoing study programmes. This will start with an annual “risk-analysis” performed by NOKUT based upon data from the national database on education100, information from audits and eventual formal complains from student organisations or other informants. The analysis will be followed by a dialogue with the relevant higher education institutions and possible demands to take corrective action. If critical matters are not rectified within a given time-period, NOKUT may conduct a new audit, start a revision of accreditation of one or more study programmes, or start a full revision of institutional accreditation.

The model will work in the following way101: NOKUT will start a dialogue whenever the probability of inferior educational quality is high and the consequences are severe, and may also do so in cases of high probability and medium severe consequences or vice versa. In special severe cases, the analysis may trigger a full revision without a foregoing dialogue phase. However, this will not be normal procedures.

The criteria underlying the process of institutional audit are not prescriptive and follow the guidelines of the ESG and other general requirements set by the Ministry of Education and Research, which establish that102:

- “Universities and colleges must have a quality assurance system that ensures continuous improvement, provides satisfactory documentation of quality work and identifies inferior quality.

- The quality assurance system must cover all processes that have a bearing on educational quality, from the information given to prospective students to the completion of study programmes. There must be routines for the students’ evaluation of the provision, the institution’s self-evaluation and its follow-up of evaluations. Documentation of the institutions’ work with the total learning environment and routines for the quality assurance of new provision must also be included.”

In the ambit of its competences, NOKUT defined the specific assessment criteria to be applied in institutional audits, which are based on two underlying principles: quality assurance systems should be geared both towards control and enhancement of the quality, and its criteria should focus on the quality of the system, and not so much on specific quality assurance methods or measures. The criteria established by NOKUT were adopted after consultation with the higher education sector.

100 All providers of higher education have to report on the number of staff and students, their study programmes and a given set of performance indicators to a public national database. Data with specific interest for quality assurance will be collected within a special “NOKUT entrance”. 101 Information provided by NOKUT (April 2011). 102 Criteria for Evaluation of Universities and University Colleges’ Quality Systems for Educational Activities, NOKUT, p. 1.

54

The criteria for the first cycle of audits focused on the structure of the quality assurance system - including the link to strategy and management and the broad participation of students and staff, the documentation the system provided and the institution’s ability to use this documentation for taking corrective measures and making priorities and allocate resources.

The first cycle of audits took place between 2003 and 2010, and the results were generally positive for the development of the quality processes within the institutions. The new cycle, which began in 2009, basically uses audit criteria that cover the same main aspects, although due to the current level of development of internal quality assurance systems, audits are now more focused on its operation and results, namely regarding quality enhancement measures taken103 and the institutions’ own use and benefit of their systems. The process is designed to be more dynamic; starting with the institution providing overarching information of the system, reports on educational quality and the leadership’s views on status and challenges within the internal quality work. Also, the institutions may suggest areas for more thorough studies in the audit. Next, the audit panel conducts a first site visit to have discussions with the institutional leadership. Then the panel decides upon eventual aspects of the quality work they want to study in some detail, and set the program for the main site visit.

In conclusion, the Norwegian system for quality assurance in higher education, which benefits from a long tradition of trust between institutions and authorities104, largely operates in a decentralized way as regards accreditation of programmes, allowing accredited institutions to self-accredit their educational offer. However, it contains effective institutional accreditation control mechanisms, as well as mechanisms for supervision and institutional and study programme accreditation revision, by attributing to NOKUT the responsibility for conducting cyclical audits of quality assurance systems of the institutions, and to pass or fail them, with direct effects on the right of self-accreditation, as well as the right to promote the review of a specific accreditation, with the possibility of revoking it in the case of a negative result.

Also worthy of note is the remarkable way in which the adoption of the Bologna reforms was effected in the national juridical framework, through an integrated approach, conducted under the motto of quality and prioritizing trust. This was a process in which the effective reinforcement of institutional autonomy and the establishment of quality assurance procedures – covering, in a balanced way, the functions of accountability and improvement in teaching – go hand in hand. 103 Froestad, W. (2009). Norway. In AQA (2009b), p. 78. 104 Ibidem, p. 77.

55

7.1.4 Network for Quality – Participation of the Universities in the Swiss model105

The federal law which regulates financial support to Swiss universities, published in 1999, establishes that, in order to have access to funding, institutions must provide high quality service, evaluated by the Centre of Accreditation and Quality Assurance of the Swiss Universities (OAQ) and recognised by the Swiss University Conference (SUC)106. As a consequence, the OAQ Agency received a mandate to promote a four-yearly summary assessment process, with the aim of confirming that universities receiving federal funding fulfil the necessary quality requirements107.

At the beginning of 2003, with the agreement of the federal government, the OAQ began to develop a process of institutional audit for the assessment to be undertaken in 2003/4. For this effect, the Agency established four standards in terms of minimum requirements for an internationally acceptable system of quality assurance, which were formulated in a fairly generic way108:

- “The quality assurance system at a university or university institution should aim both for quality control and for quality development.

- Quality assurance must be an integral part of the overall strategy of the university or university institution, cover all sub-units of the institution and be applied systematically.

- The results of internal and external evaluations as well as other quality assurance measures must be used continuously with the aim of improving quality of instruction and research.

- The effectiveness of quality assurance measures must be checked periodically by external experts.”

All public universities were subject to audit, but individual audit reports were not published, and only a general summary of the main conclusions drawn from the set of audits was made public.

The universities were not involved in the preparation of the audit processes, which was reflected in the level of acceptance of the model and was the cause of some problems for the OAQ109. In the general report on the first cycle of audits, presented in January, 2005, a final note states that “the universities, their committees and the Swiss

105 This section has been updated in late April 2011, based on a direct contact with Laura Beccari, at OAK. 106 The Swiss University Conference is a joint organization of the Swiss Confederation and of the cantons for university policy, with an essentially political composition of representatives of the federal government and of the cantons, where the President of the Swiss Rectors’ Conference has a seat as an observer. 107 Maurer, S. & Beccari, L. (2009). Quality Audits in Switzerland. In AQA (2009b), p. 62. 108 OAQ (2005). Summary Quality Assurance Audits According to the UFG – OAQ Synthesis Report, p. 2. 109 Brink, C. & Kohler, A. (2009). Audit Approaches – Features for a Common Understanding. In AQA (2009b), p. 41.

56

University Conference are to cooperate closely in drawing up quality assurance guidelines for university-level educational institutions in Switzerland”110. Swiss universities were involved in this process, within the context of the Swiss Quality Network, which led to the publication, by the SUC, of the document Quality Assurance Guidelines for Swiss Universities, accepted as a basis for the criteria to be applied in the new cycle of audits, in 2007/8.

The guidelines thus defined establish that each university is responsible for the setting up of an internal system of quality assurance and includes a set of minimum requirements, set out in seven standards which the universities must comply with. These standards follow the ESG very closely, although they contain their own specificities insofar as the process adopted covers not only teaching but also research and support services. In the handbook for universities, each of the standards is accompanied by detailed comments, to illustrate its application, and also by the indication of documentation of interest in the context of the standard. The formulation of the standards is as follows111:

- Strategy – “The University establishes its quality assurance strategy and communicates it publicly. This strategy includes the guidelines to a quality assurance system whose objective is to ensure and continuously improve the quality of university activities, as well as to promote the development of a quality culture.”

- Area of application – “The quality assurance system incorporates the university’s key tasks, especially teaching and research, as well as the interconnected supporting services. It is an integral part of the overall strategy and supports university development.”

- Processes and responsibilities – “The university regulates the quality assurance processes and ensures that its personnel and students are familiar with these processes. The responsibilities for quality and quality assurance are assigned transparently.”

- Evaluations – “The university performs periodic internal evaluations of teaching, study programmes and curricula: procedures to assess student performance, results of teaching, research and services, as well as resources, gender equality and learning infrastructure. An external evaluation is performed whenever necessary.”

- Human resources development – “The university supports and encourages the continuing education and development of its teaching and research staff. This also includes career planning for young academics and the promotion of gender equality.”

- Application of information and decision-making – “The university management bases its strategic decisions concerning research, study

110 OAQ (2005). Summary Quality Assurance Audits According to the UFG – OAQ Synthesis Report, p. 7. 111 OAQ (2007). Quality Audits 2007/08 – Concept, Procedure and Quality Standards: Guide for Universities, pp. 9-16.

57

programmes as well as the recruitment and promotion of its teaching staff on relevant and up-to-date quantitative and qualitative information. This information is systematically collected, analysed and applied in order to continuously improve the quality of university activities.”

- Communication – “A transparent reporting on procedures and results of quality assurance measures guarantees feedback to the involved groups within the university. Periodically, the university publishes objective information on study programmes and conveyed degrees.”

In the initial cycle, the audit only covered the internal quality assurance system at institutional level. In the 2007/8 cycle, apart from the quality assurance system as a whole, the mechanisms of quality assurance in teaching in a representative sample of courses were also included. The final audit reports, which included recommendations for internal quality assurance measures, began to be published by the Agency, in consultation with SUC, with the observance of data protection as established in the guidelines for quality assurance mentioned above112.

The OAQ does not foresee any formal follow-up mechanisms undertaken by the Agency in the audited units individually, but the regular meetings with the members of the Q-Network assure a continuous collective follow-up and share of experiences and good practices. Additionally, audits undertaken in 2007/8 explicitly took into consideration the recommendations made in the preceding cycle.

In brief, Switzerland belongs to the group of European countries which first adopted a model of institutional audit as the only process of external quality assurance, even though, in the first phase, it was geared almost exclusively towards the function of control, which was not completely understood or accepted by the universities. The pro-active position adopted by the latter and the communication and dialogue between the OAQ and the universities, facilitated by the establishment of a ‘Swiss University Quality Network’, were a vital component for the review of the model, which began to have more acceptance on the part of the institutions, and reinforced the proposal for continuous enhancement of quality.

7.1.5 Postponement of audits – A backward step in Denmark

Denmark was one of the first countries to establish a national system for external assessment of higher education, when in 1992 it created the Danish Centre for Quality Assurance and Evaluation of Higher Education, which operated until 1999, when it was substituted by the Danish Evaluation Institute (EVA), with a scope of action which was

112 Maurer, S. & Beccari, L. (2009). Quality Audits in Switzerland. In AQA (2009b), p. 64.

58

extended to all levels of teaching, from Early Childhood Education to Adult Education and Continuing Training113.

Institutional audit was established in 2003 by an agreement between the EVA Agency and the Danish Rectors’ Conference114. In the terms of this agreement, all universities agreed to undergo quality audit, with the timing of the audit to be decided by each university. Thus, annually the Agency extended an invitation to the institutions to voluntarily register for audit in that particular year. As would be expected, given the genesis of the process, the institutions were keen to take part.

The explicit aim of the adopted model for the audits is to provide support to the institutions in the area of quality and quality enhancement, and is thus much more focused on improvement than on control. Basically, assessment of the internal quality assurance system is undertaken in order to identify its strengths and weaknesses, as well as the efforts made by the institutions for its improvement. Recommendations are given on ways for a more systematic organization of work in the area of quality.

In terms of standards for these audits, twelve criteria were established, formulated as declarations which describe a well-developed internal system of quality assurance. The standards are in accordance with the ESG, itemising them in some cases.

To help in the preparation of the self-assessment report, the Agency prepared guidelines aiming to encourage internal reflection on the strengths and weaknesses of the quality assurance system and its potential improvement, and to facilitate the organization of information to be included. These guidelines are presented by the Agency at an initial seminar, which takes place at the institution to be audited.

The results of the audits have no formal consequences and it is up to each institution to decide what action to take regarding the report and the recommendations. The EVA Agency did not establish any follow-up mechanisms, but in a number of cases seminars or conferences were organized as part of the follow-up activities undertaken, on the initiative of the institution, with the participation of members of the Agency and members of the audit team.

As a result of the experience gained in the audits, the EVA Agency reported serious efforts on the part of universities to develop internal quality assurance systems and also that the audits contributed to increase focus on quality. The main challenges facing universities were identified as ensuring better coherence and more systematic approaches in quality work, in order to avoid the frequently seen situation in which the 113 History in Short, EVA. http://english.eva.dk/about-eva/history-in-short. Retrieved 09.12.2009. 114 Enroth, I. (2009). Audits in Denmark. In AQA (2009b), p. 95.

59

quality assurance mechanisms are not sufficiently linked to the core academic activities. Better use of quality assurance results in the development of the institution was also mentioned115.

The process of institutional audit was in use between 2003 and 2007, but was then suspended due to a political decision to introduce programme accreditation as a formal process of external quality assurance, and a new Agency responsible for accreditation of university programmes – ACE DENMARK – was created. EVA states that it is interested in maintaining the audit model as a relevant method to assure quality at institutional level, and in 2009 initiated a pilot auditing scheme in the polytechnic sector, which had shown interest in the process as a means of developing its quality work116.

Whatever the case, this recent evolution in the Danish situation goes against current trends in quality assurance and is a good example of how external pressure, namely on the part of political powers, can influence quality assurance systems, in a move towards greater external control of quality in institutions. This is particularly notable when these systems were initially very strongly geared towards quality enhancement, though they had perhaps not paid enough attention to providing public information on quality within the institutions.

7.1.6 Respect for autonomy – The Finnish model117

From the outset, the system of quality assurance in Finland was quite open and receptive to validation of different types of external assessment carried out by the institutions, namely international assessments. In 2004, after a national period of reflection, it was proposed that all universities and polytechnics should develop quality assurance systems in relation to all their activities and that they should be regularly audited by the Finnish Higher Education Evaluation Council (FINHEEC). The audit model was decided upon in 2005, on the basis of voluntary participation on the part of institutions, but all of them, universities and polytechnics, adhered to the process, agreeing to an audit process which will last until 2011118.

The national system of quality assurance in higher education includes three interrelated components:

115 Ibidem, p. 98. 116 Ibidem, p. 95. 117 This section has been updated in May 2011, based on a direct contact with Kirsi Mustonen, at NOKUT. 118 Moitus, S. & Seppälä, H. (2009). Audits of Quality Assurance Systems of Finish Higher Education Institutions. In AQA (2009b), p. 66.

60

- National higher education policy defined by the Ministry of Education and Culture, which negotiates with each institution a triennial agreement on objectives and performance;

- The institutions’ own quality assurance; - National auditing, performed by FINHEEC.

The design of the auditing process is rooted in a basic assumption, respecting institutional autonomy: the audits do not touch upon the objectives, operative contents and performance of the institutions, per se, as the assessment of results is the responsibility of the institutions themselves, in the ambit of its three-yearly contract with the Ministry. The main aim of the auditing process is to enhance the quality of higher education, and to provide support to institutions in the development of their internal quality assurance systems, to identify and disseminate good practices and to encourage their adoption by the institutions, and to promote debate on quality and on the interaction between institutions and stakeholders119.

The specific aims of the Finnish auditing process are: - To establish the qualitative objectives set by the HEI for its own activities; - To evaluate what procedures and processes the HEI uses to maintain and

develop the quality of its education and other activities; - To evaluate whether the HEI’s quality assurance works as intended, whether

the QA system produces useful and relevant information for the improvement of its operations and whether it brings about effective improvement measures.

The audit focuses on two different levels of analysis: that of the comprehensiveness, performance, transparency and effectiveness of the quality assurance system, and also the meta-evaluation of the system, i.e., the way in which the institution monitors, assesses and develops its own quality assurance mechanism. For this effect, seven auditing targets were defined120:

1. “Definition of the objectives, functions, actors and responsibilities of the HEI’s QA system as well as the respective documentation.

2. The comprehensiveness and effectiveness of the QA procedures and structures related to the HEI’s basic mission:

a) Degree education; b) Research/R&D121; c) Interaction with and impact on society as well as regional

development co-operation; d) Support services (such as library and information services, and

international services); e) Staff recruitment and development.

119 FINHEEC (2007). Audits of Quality Assurance Systems of Finish Higher Education Institutions, p. 10. 120 Ibidem. 121 In March 2009 FINHEEC added Artistic Activities to this target.

61

3. Interface between the QA system and the management and steering of operations.

4. Participation of HEI staff, students and external stakeholders in quality assurance.

5. Relevance of, and access to, the information generated by the QA system:

a) within the HEI; b) from the perspective of the external stakeholders of the HEI.

6. Monitoring, evaluation and continuous improvement of the QA system. 7. The QA system as a whole.”

The scrutiny of the auditors is based on a set of criteria which describe different stages of development in relation to each of the target areas and sub-areas, expressed on a four-level scale: ‘absent’, ‘emerging’, ‘developing’ and ‘advanced’. Thus, a matrix ‘target areas vs stages of development’ is defined, which specifies the criteria for each pair ‘area /stage of development’122.

The performance of an audit implies a previous formal agreement between the institution and the FINHEEC Agency, establishing the way in which the audit will be carried out, its timetable, the number of foreign experts in the team, the language to be used, the duration of the audit visit, the division of costs and the commitment on the part of the institution to accepting a re-audit, if necessary. The Agency provides detailed guidelines concerning documentation to be submitted by the institution, which includes basic material on the quality system, and evidence and examples to substantiate the performance of the system. This is primarily based on pre-existing materials, some of which may be accessed on the internet page of the institution.

About three weeks before the actual audit, and as another way of helping the institution to prepare for the event, the chair of the external team, and the representative of the Agency who is coordinating the audit, visit the institution. During this visit a seminar, open to all staff and students, is organised with the aim of explaining how the process works and to promote debate and provide clarification.

The findings of the audit are expressed in terms of a positive decision regarding certification of the quality assurance system, or the need for a re-audit, to be carried out around two years later, in accordance with specific terms of reference, focused on the improvement proposals made. For a positive result to be achieved, it is necessary that all target areas meet the minimum requirement of ‘emerging’, and that the appraisal of the system as a whole is at least ‘developing’. The positive appraisal is published on the web page of the institution and a certificate is awarded, which is valid for six years. 122 Ibidem, pp. 29-31.

62

Apart from the audit result, the final report includes recommendations concerning aspects that the team considers may be improved. As well as being published by the Agency, the report is presented at a seminar organized in collaboration with the audited institution, with the participation of the external team, where there is open discussion of the results of the audit.

The Agency also organises a follow-up seminar to support continuing development of the quality assurance system within the institution, which takes place approximately three years after the audit decision, and which is open to all higher education institutions, thus contributing to the exchange of experience and best practices.

The feedback from FINHEEC is very positive regarding the impact of the audits on the development of systematic internal quality management procedures, on a greater participation of the different stakeholders, including students, and also on the visibility of the policies, criteria and quality indicators applied by the different institutions123. Because of the way it is organized, with total respect for institutional autonomy, in an atmosphere of dialogue and constructive collaboration with the institutions, the Finnish model is undoubtedly a reference in the European context.

7.1.7 ‘System Accreditation’ – A new model for accreditation in Germany

In Germany, responsibility for higher education, including quality assurance, is directly in the hands of the sixteen federal states (Länder). The Conference of Ministers of Education and Culture (KMK) promotes some alignment of policies between the different states, although its deliberations are not binding and only affect the higher education institutions after becoming policies and regulations of the respective federal state.

Until 1998, there was very strong state control over the creation of programmes, exercised by the ministries of the federal states, based on national frameworks of reference for curricula by area of knowledge, with the aim of ensuring common standards and comparability of degrees in the whole of Germany. Thus, ex-ante control of the quality of teaching was exercised, and there was no national quality assurance system. However, in some institutions and states various teaching assessment procedures were being introduced, with a view to increasing transparency, reinforcing institutional responsibility, supporting institutions in the introduction of systematic measures of quality enhancement and promoting the profile, image and competitiveness

123 Moitus, S. & Seppälä, H. (2009). Audits of Quality Assurance Systems of Finish Higher Education Institutions. In AQA (2009b), p. 70.

63

of higher education in Germany124. These initiatives, strongly influenced by the pilot project of the European Commission undertaken in 1994/5, and largely based on a resolution by the German Rectors’ Conference125 (HRK), were, however, far from being generalised.

The legal reforms introduced in 1998 made it possible for the institutions to create new programmes corresponding to Bachelor and Master degrees, for which the previous system of state control of curricula was substituted by a process of accreditation, with the aim of allowing more curricular innovation and a greater diversity of educational offer, providing the institutions greater autonomy in the organisation of their teaching.

Through an agreement between HRK and KMK (and not by law), the German Accreditation Council (GAC) was created, as an autonomous national structure, responsible for the establishment of comparable quality standards for the new Bachelor and Master degrees. These standards were to be applied through a decentralized model of accreditation, in which the processes of accreditation are undertaken by private, independent agencies, accredited for the effect by the GAC.

Currently, there are seven agencies, four are sectorial126 in nature and three have a regional focus127, operating on an openly competitive basis, i.e., the institutions have free choice of the Agency they wish to use for the accreditation of their courses. Two foreign agencies are also accredited by the GAC – the AQA (Austrian Agency for Quality Assurance) and the OAQ (Centre of Accreditation and Quality Assurance of the Swiss Universities). They can consequently undertake accreditation of higher education programmes in Germany with official validation.

In spite of having had the initial agreement of the HRK, the model for accreditation of study programmes, thus created, progressed rather slowly128 and led to a generalized wave of dissatisfaction among higher education institutions and other relevant actors at a political level. There were two main types of criticism. On one hand, it was pointed

124 Erichsen, H.-U. (2004). Quality Assurance and Accreditation in Germany. In Di Nauta et al. (2004), p. 42. 125 HRK (1995). Resolution: Evaluation in German Higher Education Institutions with particular reference to the evaluation of teaching. 126 AHPGS - Accreditation Agency for Study Programmes in Health and Social Sciences AKAST - Agency for Quality Assurance and Accreditation of Canonical Study Programmes

ASIIN - Accreditation Agency for Degree Programmes in Engineering, Informatics/Computer Science, the Natural Sciences and Mathematics

FIBBA - Foundation for International Business Administration Accreditation 127 ACQUIN - Accreditation, Certification and Quality Assurance Institute

AQAS - Agency for Quality Assurance by Accreditation of Study Programmes ZEvA - Central Evaluation and Accreditation Agency Hanover

128 According to data from the HRK, in 2007 only 37% of new programmes were accredited.

64

out that there was great expense and effort involved, as institutions had to submit all new programmes to a process of accreditation, which was expensive and burdensome in terms of administration, and they were required to pay the respective costs themselves (which came to over ten thousand euros per programme to be accredited, in fees to the agencies). On top of this, various institutions considered that the accreditation of study programmes was interfering too far in institutional autonomy, in spite of the initial expectation that the new process would bring more autonomy concerning curricula and that it might stimulate innovation and diversity. In reality, the imposition of the accreditation criteria by the agencies, together with the fact that in the majority of federal states the respective ministries submitted the dossier for the creation of new courses to a formal approval process, following the accreditation process, was seen by many as a control over the curricula which was just as tight, if not more so, than before.

Due to this climate of concern and contestation, a pilot project was launched in 2004 led by the Agency ACQUIN in cooperation with the HRK, two universities (Bayreuth and Bremen) and two polytechnic institutions (Erfurt and Münster), which had the financial support of the federal ministry, with the aim of developing and testing a new methodology known as ‘processaccreditation’, which focused on the verification and validation of internal processes with which the institutions assured the quality of their programmes129. The project had two distinct but interrelated objectives130, which were: to optimize the quality of the teaching and learning processes, i.e., the development, implementation, monitoring and improvement of programmes, and the development of a system to accredit quality processes in teaching and learning.

The Project ran between 2004 and 2006, and influenced a large number of measures taken the following year. Indeed, in April 2007, the ministers of three federal states131 made a joint statement demanding the introduction of a new accreditation process, similar to the accreditation of processes, as an alternative to accreditation of programmes, threatening to abandon the national system in force if progress was not seen in this area. The KMK, with the support of GAC, reacted positively to this pressure and, in June 2007, decided to open the door to the possibility of installing a process of accreditation of internal quality assurance systems for teaching and learning in higher

129 Witte, J. (2008). The Changing Political Framework of Quality Assurance in German Higher Education: National Debates in the European Context. In EUA (2008), p. 50. 130 ACQUIN (2004). Pilot Project “Process Quality for Teaching and Learning – Concept and Implementation of a Process Accreditation Approach”, p. 2. 131 The Länder of Baden-Wurttemberg, Bavaria and North Rhine-Westphalia.

65

education institutions, known as ‘system accreditation’132. This was to operate as an alternative to the model which accredited individual programmes, leaving it up to the institutions to choose either of the models. The reasons invoked by the KMK for such a significant change in policy were, on one hand, to decrease the time and effort invested in accreditation by the institution, while at the same time providing evidence of the trustworthiness of the internal quality system. The other reason was to reinforce the autonomy and responsibility of each higher education institution in ensuring the quality of their programmes.

After preparation and publication of the general criteria for system accreditation by the GAC, the KMK gave authorization for the new process of accreditation to be used as from 2008 by the agencies that, after submitting themselves to a review of the criteria they used, obtained accreditation from GAC, for the effect. The two accreditation systems have been operating in parallel, a situation which is likely to continue.

The system accreditation specifically promotes the assessment of structures and processes relevant for teaching and learning, with regard to their suitability for the attainment of the objectives of the different qualifications, and to guarantee a high level for the programmes, and also to the degree of compliance with the ESG, the guidelines of the KMK and the criteria established by the GAC. In order for an institution to be able to apply, for the fist time, for this process of accreditation it is required that it has at least one accredited programme for each 2,500 students enrolled, always including 1st and 2nd cycle programmes. In cases where the institution operates old programmes which were regulated by the state, one or more of these programmes at each level should be accredited133. A positive result in system accreditation implies that all of the institution’s programmes, including those created after accreditation, are automatically considered accredited.

The criteria established by the GAC for system accreditation, which are formulated in a fairly open manner, are organized in six standards, systemized as follows134:

- Qualification objectives – “The Higher Education Institution has defined and published an education profile for itself as an institution and for its study programmes as part of a strategic development concept. It has and is continuously using methods to check the qualification objectives of its study programmes.”

132 The names “process accreditation” and “system accreditation” were used interchangeably for some time but the latter, being more generic, was generally adopted due to the fact that quality assurance does not only concern processes but also objectives, structures and results. 133 GAC (2008). Criteria for System Accreditation, Printed Matter 11/2008 (resolved on 08 October 2007, amended on 29 February 2008), p. 1. 134 Ibidem, pp. 2-4.

66

- Management system in the field of teaching and learning – “The Higher Education Institution has and is continuously using a management system in the field of teaching and learning. This ensures determination of concrete and plausible qualification objectives of study programmes, taking into account the ‘Criteria for the Accreditation of Study Programmes’ as amended in its latest version. The qualification objectives include disciplinary and interdisciplinary aspects, especially scientific competency, employability (competency to take up a qualified employment), qualification to commitment in the realm of civil society and personality development.” (the standard includes a series of detailed specifications regarding the design and implementation of programmes, their conformity with the German qualifications framework and the involvement of internal and external stakeholders, in the design and review of the study programmes).

- Procedures of internal quality assurance – “The Higher Education Institution has in place quality assurance methods in the field of teaching and learning meeting the requirements of the European Standards and Guidelines for Quality Assurance in Higher Education and integrated into an overall concept. The internal quality assurance system has staff and equipment resources that ensure sustainability. It is appropriate to assess the effectiveness of internal management processes in the field of teaching and learning and to warrant the ensuring and continuous improvement of the quality of teaching and learning. In detail, the internal quality assurance system includes the following: • Regular internal and external evaluation of study programmes, taking into

account the organisation of studies and examinations; • Regular course evaluation of the quality of teaching units by students; • Verification of the competency of professors in the fields of teaching and

examination when they are hired and their regular training advancement; • Regular verification of the compliance with guidelines for the accreditation

of study programmes specified by KMK and the Accreditation Council; • Binding procedures for implementing recommendations and a system of

incentives.”

- Reporting system and data collection – “The Higher Education Institution has an internal reporting system that documents the structures and procedures in the development and implementation of study programmes as well as the structures, procedures and measures of quality assurance, its results and effects.”

Responsibilities – “The decision procedures, competencies and responsibilities within the management system for teaching and learning and within the internal quality assurance system are clearly defined and published throughout the Higher Education Institution.”

- Documentation – “The Higher Education Institution informs the bodies in charge of teaching and learning at least once per year and in addition the general public and the responsible ministry in an appropriate manner about the procedures and results of the quality assurance measures in the field of teaching and learning.”

A system accreditation exercise involves three main components:

67

- Assessment of the institution’s management and quality assurance system; - An examination of the efficacy of the institution’s management and quality

assurance system, undertaken through: • in a horizontal approach to all study programmes, a random sample of

features relevant to planning, implementation and assurance of the quality of the programmes135, to ensure that their implementation and regulation (does not touch contents) comply with the specific KMK guidelines and the GAC criteria for accreditation of study programmes;

• in a vertical approach, an in-depth analysis of a random sample (of around 15%) of programmes, focusing on study objectives, organization, contents, resources, implementation of procedures and aspects of quality assurance related to the programme, which does not, however, lead to an individual accreditation decision for these programmes, or otherwise.

The process involves three visits to the institution, the last of which, concerning the assessment of the selected programmes, is undertaken by teams which are different from that which undertakes the global assessment of the system.

There is also a compulsory formal follow-up process, according to which the institution has to submit a sample of study programmes to external assessment in the middle of the accreditation cycle (i.e., four years after accreditation). The institution has free choice of Agency for the assessment, which may be different from the Agency which carried out the system accreditation. The only purpose of this intermediate assessment is to give feedback to the institution, and it may contain recommendations for correction of areas where quality was found to be lacking. The report is, however, published by the Agency.

The ACQUIN, which had been involved in the pilot study, was the Agency best prepared to begin system accreditation at once (which was something of a problem in a decentralized accreditation system which is supposed to be truly competitive). The handbook adopted for the implementation of the process follows the GAC guidelines and criteria very closely, although it goes into more specific detail about the points to be considered in relation to each standard136. There is a good deal of interaction with the institution to be accredited during the accreditation process, which involves the following stages: a preliminary conversation to agree on accreditation objectives; the submission of preliminary documentation in order to verify if the institution fulfills the

135 The ACQUIN Agency has identified a catalogue of components considered important due to their influence on the quality of study programmes, from which those to be included in the sample are generally selected. The catalogue includes, for example: the definition of qualification objectives; the compliance with the framework regulations for the introduction of an ECTS credit system and the modularization of study programmes; the student work load; the organization and coordination of studies; or the general and subject-related student advisory services (ACQUIN, 2009, p. 8). 136 ACQUIN (2009). Guidelines for System Accreditation Procedures.

68

pre-conditions of admission to system accreditation; the drawing up of a contract for the accreditation process, which includes the scope of the exercise, the time frame and costs; the preparation and submission of relevant documentation; the assessment of the documents by a team of (at least five) experts; on-site visits; the compiling of the evaluation report, which the institution may comment on, indicate corrections to be made and describe measures for improvement initiated in the meantime; the presentation of the report by the team of experts to the accreditation Agency, with a recommendation regarding the decision to be taken; the final decision.

Other agencies then began to work on their own ways of applying the criteria defined by the GAC, such as for example the ZEvA Agency, which also began to offer a model of institutional assessment based on an auditing process. Since October, 2008, the AQAS Agency has also been qualified to undertake system accreditation.

Thus, it was in this way that the German system of quality assurance sought to respond to the challenge of the need to create methods which would be less intrusive and less costly in terms of resources than the accreditation of study programmes, but which are at the same time at least as effective. System evaluation continues, however, to be a formal accreditation procedure, with a binary decision (accreditation, no accreditation), and still contains an element of evaluation at study programme level, although only regarding a random sample of 15% of the programmes offered137. Even so, it is a movement in the direction pointed out by the Bologna Process, towards greater autonomy and responsibility of higher education institutions. However, concerns are noted by some observers, namely regarding the embryonic stage of development of internal quality systems in the majority of institutions at the time the changes were made, with the consequent risk of there being essentially formal cosmetic adequation to external criteria, with little impact on the quality culture within the institution. In this regard, it is interesting to note that before the introduction of the Bologna Process, there was an increase in the number of initiatives of internal and external assessment, of various types, geared towards quality enhancement, which were not duly considered at the time of the debate on quality assurance of Bachelor and Master degrees, which was centred almost exclusively on the external accreditation process, which is now seen as a mistake138.

137 Hopbach, A. (2009). Germany. In AQA (2009b), p. 86. 138 Witte, J. (2008). The Changing Political Framework of Quality Assurance in German Higher Education: National Debates in the European Context. In EUA (2008), p. 51.

69

7.1.8 Conciliating support and audit– The Austrian Quality Assurance Agency (AQA)

The Austrian higher education system has a very particular type of organization, in that each sector is regulated by its own specific law, and there is no common regulating framework139.

Public universities and public teacher training schools are created by law and are not subject to any obligatory form of institutional or study programme accreditation. They do, however, have to respect multiannual performance contracts signed with the Federal Ministry of Science and Research, with a three year scope, which include the commitment of the institution regarding quality objectives and measures, for which they must supply evidence of effective achievement at the end of the contract period.

Public polytechnic institutions are subject to an initial accreditation process of their study programmes undertaken by the Austrian Fachhochschulen Council. Private institutions, in turn, are obliged to go through a process of institutional accreditation and initial accreditation of study programmes for which the Austrian Accreditation Council is responsible. Accreditation was thus introduced as a guarantee of pre-defined quality standards in the most recent sectors of Austrian higher education. Accreditation is valid for a limited period and re-accreditation decisions are taken on the basis of the results of external assessment, undertaken by the Austrian Agency for Quality Assurance (AQA) or by other internationally recognized agencies.

All institutions, whatever sector they belong to, are obliged by law to take responsibility for their own quality and to install an internal quality assurance system. However, no specifications are defined for the structure of the internal systems, and it is up to each institution to choose the instruments and procedures to apply, as well as the structure and competences of the internal units created for this end. In practice, the majority of Austrian higher education institutions have been establishing internal systems to deal with quality and assessment, creating specific units for the effect, involving students in the assessment of teaching and, more and more, also involving their graduates, often through alumni associations140.

The implementation of internal systems of quality assurance has not, however, gained much prominence in external assessment procedures associated with accreditation, representing just one among many in a large spectrum of evaluation criteria. There would appear to be a similar situation regarding the formal performance

139 Kohler, A. (2009). Quality Assurance in Austrian Higher Education – Features and Challenges. In Bozo et al. (2009), p. 9. 140 Ibidem, p. 10.

70

reports presented by the institutions141. In view of the performance agreements established with the government, it is likely that it will become more urgent for the institutions to present evidence of the efficiency of their institutional quality management, obtained in an independent way from external procedures.

The law does not foresee any specific mechanism for the presentation of this evidence, but the most credible way of obtaining it, with external validation, is through external certification of quality assurance systems. The AQA Agency rose to this challenge and developed a certification service based on audit procedures, which are formally recognized by the federal government, and to which the institutions may voluntarily subscribe to.

However, the certification process will only be viable when based on real evidence of performance of internal quality systems, which requires that these systems have been installed and operating for some time. Before they reach this stage, it would be important for the institutions to be able to avail themselves of support for their organizational development with regard to the management and promotion of quality. The scheme developed by the AQA took this into account and offered a service with two features which are complementary, but at the same time totally independent of each other:

- The provision of specialised help for institutions who wish to establish and document their internal quality assurance system in one or more specific performance areas;

- The certification of an established internal quality assurance system, in a specific performance area or for the whole institution.

The performance areas covered, which correspond to the multiannual contracts signed with the federal government, are teaching/learning and continuous training, research and development, internationalization and mobility, and management and development of human resources, which, as has been mentioned, can be dealt with individually or together, both regarding the support and the certification functions142.

The model is built along three lines which are operationally independent of each other:

- Expertise and Support, with a view to reinforcing the ability of an institution to adapt its steering processes to its quality objectives, and to implement the processes of monitoring and managing quality in relation to a specific performance area selected by the institution;

141 Ibidem, p. 11. 142 Hanft, A. & Kohler, A. (2008). How Can External Quality Assurance Support Institutional Quality Management. In EUA (2008), p. 54.

71

- Focus Audit, which makes available a quality audit process and the certification of the quality management mechanisms, focused on a specific performance area chosen by the institution;

- Advanced Audit, which offers auditing of the quality and certification of the system of quality management of the institution as a whole.

The institutions can freely opt for either of the functions offered by the model, that is, none of the features constitute a pre-condition for the others.

The Support and Consultancy function is carried out over a period of two years with the help of specialists selected by the Agency, with the agreement of the institution. The process involves the following steps:

- The definition of quality objectives and the development of internal quality assurance mechanisms;

- The description and explicit documentation of objectives and strategies for quality;

- An analysis of the steering and management tools in terms of their contribution for the accomplishment of the quality objectives;

- The carrying out of a duly monitored pilot-experiment of application of the mechanisms chosen, with subsequent adaptation or correction of the tools and their possible transfer into other performance areas.

The result of this exercise is the implementation of quality management processes in the selected area, including the internal monitoring procedures.

The AQA reports positive results of this experiment, namely regarding the creation of an internal impetus concerning quality processes following the initial steps taken with the support of external experts, and the resulting involvement of members of the institution143.

The Focus Audit implies that an institution voluntarily puts itself forward for external assessment of its quality management system, in a performance area of its own choice, with a view to determining the respective degree of success in the achievement of the quality objectives and strategies defined by the institution. The external audit focuses on two key processes in relation to the selected area, agreed upon by the institution and the AQA.

The audit process involves the preparation of a structured, self-assessment report, an external assessment with two visits to the institution, and a follow-up process. As far as possible, the self-assessment report is based on documentation which already exists in the institution, which should explicitly indicate the quality objectives to be achieved, the

143 Ibidem, p. 55.

72

resources to be mobilised, the means of monitoring the system and the people responsible.

The assessment focuses on two distinct levels – the assessment of the organization and structure of the quality management system, and the assessment of evidence of the effectiveness of the operation of the system - and it is undertaken with reference to six standards specified by the AQA in accordance with the ESG144:

1. “The HEI possesses a strategy for quality and integrates this strategy into its steering mechanisms.

2. Quality assurance and development are a part of the leadership’s responsibilities.

3. The HEI clearly defines and makes transparent the assignments of responsibilities and duties, and it also provides easy access to information needed for achieving its goals.

4. Procedures for internal quality assurance are established and the quality management system supports the core processes of the HEI.

5. Monitoring and information systems are an integral part of the quality management system and facilitate regular reporting.

6. A quality management system includes the systematic participation of the different stakeholders.”

The external assessment panel is mainly composed of international experts, includes a student, and cannot include specialists who have been involved in the Support and Consultancy phase, if there was one. The panel members produce a report in which they express their appraisal of the stage of development of the quality management system in the respective performance area in relation to each of the six standards (on a four-level grid ‘not visible’, ‘in development’, ‘considerable development’ and ‘advanced system’), and can also include recommendations for improvement and development of the system. A positive assessment, which implies that none of the standards were classified as ‘not visible’, and that not more than one standard was classified as ‘in development’, leads to certification of the system by the AQA, in the performance area assessed.

If institutions request Advanced Audit, external assessment includes the four performance areas and covers four different key aspects, one in each area. If installed procedures are judged to be effective in all areas, regarding all six standards mentioned above, the AQA certifies the quality management system as a whole.

Certification is awarded for a period of six years and, in the case of public universities, can be used before the federal ministry as evidence of fulfillment of the

144 AQA (2009a). AQA Quality Audits – Accreditation of the Quality Management of Performance Areas at the Higher Education Institutions, p. 4.

73

relevant parts of the performance contract established. In this case, institutions are exempt from submitting reports on aspects which are at least in part covered by the certification. The Agency hopes that polytechnics and private institutions may come to use certification as a way of demonstrating that they have fulfilled the legal requirement to install quality management systems145.

In conclusion, the particular way of regulating universities by the Austrian State created opportunities and encouragement for an innovative approach by AQA, which is recognized and accepted by public authorities and seen with favourable eyes by the institutions. Indeed, in a position taken in December, 2007, Austrian public universities publically and formally manifested their support and receptiveness to this type of external assessment, which, in their opinion, limits the costs of external assessments and respects the ability of the institutions themselves to deal with quality assurance146.

An important point arising from this project is that the development, implementation and validation of an institutional quality assurance system requires time – the period of two years dedicated to Support and Consultancy should be noted. The viability of institutional approaches to quality should, therefore, be established over a considerable period of time.

7.1.9 Earning Trust –Tthe case of the Netherlands and Flanders

The mission of the Accreditation Organisation of the Netherlands and Flanders (NVAO) is the (compulsory) accreditation of all higher education programmes in the Netherlands and Flanders147. The NVAO is currently conducting a first cycle of assessment, within which all study programmes in the Netherlands will undergo a process of accreditation by 2010, and in Flanders by 2013.

However, in view of the critical stance of higher education institutions, which have complained that “accreditation focussed too much on procedures and processes not directly relevant to education, was not cost-effective and was felt to be too bureaucratic”, a position which gained some echo with the government and with students148, and also due to the conviction that a second cycle of accreditation,

145 Hanft, A. & Kohler, A. (2008). How can External Quality Assurance Support Institutional Quality Management. In EUA (2008), p. 56. 146 Kohler, A. (2009). Quality Assurance in Austrian Higher Education – Features and Challenges. In Bozo et al.(2009), p. 12. 147 This Agency was established in June, 2002 as the Netherlands Accreditation Organisation (NAO), but in Setember, 2004, through an agreement between the governments of the Netherlands and of Flanders, it became known as (NVAO), also covering higher education in Flanders. 148 van Galen, S. & Woutersen, M. (2009). Balancing Quality Enhancement and Accountability – Reforming the Dutch and Flemish Accreditation System, p. 1.

74

undertaken on the same lines as the current one, would not bring enough added value to justify the financial and administrative costs involved, the Agency is preparing and testing a new accreditation model in syntony with the policy adopted by the Dutch government in 2007. This policy establishes that higher education institutions should be given greater margin for manoeuvre in the organisation of their teaching, through increased confidence in their professionals, the reduction of rules and regulamentation, and a lightening of supervision. The underlying idea is to selectively make the process of study programme accreditation less onerous, on the basis of ‘earned trust’, with the external quality assurance procedures becoming ‘made-to-measure’, with intervention of the Agency whenever necessary, but giving the institutions space when their internal quality mechanisms have earned trust. The NVAO considers that in this way, “a new balance will be created between accountability, quality improvement and trust”149.

The central element of the new system continues to be accreditation of study programmes, but institutions will be able to opt for one of the two ways:

- To voluntarily submit to institutional audit as a condition for later undergoing study programme accreditation, on the basis of a simpler assessment process;

- To submit to the normal process of accreditation on similar lines to the preceding cycle.

This new approach, which was agreed upon by the government, the institutions and other stakeholders, after a long process of consultation, aims to determine – through institutional audit – to what extent the institutions themselves control the quality of their teaching, within the understanding, that had unanimous support of the different stakeholders, that the balance between ‘continuous enhancement’ and ‘accountability’ should be restored, that the system should focus on programme contents and not so much on procedural aspects, that there was urgent need to reduce formalities and that ownership of the system by the academics should be reinforced150.

The audit procedures follow the habitual model of preparation of a critical self-assessment report and of an external assessment by an independent panel, with five standards as a reference, which may be expressed in the following questions151:

- Vision – “What is the vision of the institution regarding the quality of the education it provides?”

- Policy – “How does the institution intend to realise this vision?”

149 Klaassen, L. (2009). NVAO: Introducing, developing and implementing a new phase in the accreditation system in the Netherlands and Flanders. In EUA (2009a), p. 38. 150 van Galen, S. & Woutersen, M. (2009). Balancing Quality Enhancement and Accountability – Reforming the Dutch and Flemish Accreditation System, p. 1. 151 Ibidem, p. 3.

75

- Results – “How does the institution measure the degree to which this vision is realised?”

- Improvement policy – “How is the institution working towards effecting improvements?”

- Organisation and decision-making structure – “Who is responsible for what?”

The result of the audit is expressed on a three-level scale: trust, conditional trust, no trust152.

For institutions that obtain the statute of trust with regard to their internal systems of quality assurance, the assessment of study programmes follows a lighter process based on a self-assessment report of programmes prepared by the institution and on a discussion with the external assessors, by area of knowledge, regarding aspects which are not covered by the institutional audit, translated into three standards which are specific for each programme and expressed in the following questions:

- What are the programme’s intended learning outcomes? - How does the programme intend to realise its objectives? - What are the achieved learning outcomes?

The result of external evaluation of study programmes is expressed on a four-level rating scale:

- Insufficient – the programme does not meet the generic quality requirements, that is to say, it does not reach the quality level that can be reasonably expected of a higher education programme of the same nature in an international perspective;

- Sufficient – the programme meets the generic quality requirements; - Good – the programme exceeds the generic quality level; - Excellent – the programme exceeds the generic quality level by far and may

serve as a role model for other relevant programmes.

In order to test the model, pilot projects were launched involving six institutions in the Netherlands and three in Flanders, selected in order to cover various profiles, including private institutions. The audit phase was concluded towards the end of 2008 (five institutions were awarded the mention ‘trust’, and four were awarded the mention ‘conditional trust’). This exercise was accompanied by an observer from the Inspectorate of Education in the Netherlands, at the request of NVAO, to ensure independent assessment of the audit processes. The pilot projects then progressed to the phase of simplified assessment of study programmes, to test the methodology. The first impressions gathered from the audit teams and the institutions assessed were quite positive (ranging from favourable to enthusiastic) 153.

152 Klaassen, L. (2009). NVAO: Introducing, developing and implementing a new phase in the accreditation system in the Netherlands and Flanders. In EUA (2009a), p. 39. 153 Ibidem, p. 40.

76

To conclude this brief overview, it is interesting to note that Netherlands began by adopting a model of external quality assurance clearly geared towards quality enhancement and respect for institutional autonomy, which was considered a model at the beginning of the 90’s and which influenced other national systems, including Portugal, but which, due to external pressure, as was seen in point 5.3, became an example of an undesirable change towards a main concern with ‘accountability’. The alterations now planned for the next assessment cycle represent some inversion of this process, creating more opportunities for the exercise of institutional autonomy and for approaches which are more focused on quality enhancement. The compulsory nature of study programme accreditation remains, although there exists the possibility of alternative processes which will, in principle, be simpler for those institutions which demonstrated the ability to gain the trust of the Agency and of society.

7.1.10 The Audit programme - Spain

The systematic application of quality assurance procedures in the Spanish higher education system began in 1995, with the creation of the National University Quality Assessment Plan, which aimed to encourage institutional assessment, provide assessment methods that were compatible with the developments underway within the ambit of the European Union, and to provide objective information to support decision taking by different organisms relevant to higher education. This plan was developed between 1996 and 2000 coordinated by the Council of Universities, and focused on the University Quality Plans, in which the universities expressed their quality objectives154, defined the actions to be undertaken to achieve these aims, the indicators to be used for accompaniment and monitoring of these programmed actions, their established targets and the organs and structures responsible for their execution. The II University Quality Plan was launched in 2001, running until 2003 with an increasing participation of the Autonomous Communities, which led to the creation of agencies in some of them155.

The National Agency for Assessment of Quality and Accreditation (ANECA) was created towards the end of 2001, which, in articulation with the Autonomous Community Agencies, was to take responsibility for quality assurance regarding teaching, university degrees, study programmes and teaching staff, as well as for the institutions as a whole. In 2007, the General Conference of University Policy was set up

154 Basically corresponding to the standards for teaching that the ESG refer to. 155 This historical note is based on a paper presented by the Director of ANECA at the 3rd European Quality Assurance Forum, with the reference below: Dalmau, G. R. & González, C. R. (2009). ESG and the Current QA Trends in Spain: more than just a change. In Bozo et al. (2009), pp. 14-18.

77

to promote articulation between ANECA and the regional agencies156, in order to regulate the conditions of cooperation and mutual recognition between agencies.

Following the creation of ANECA, the Institutional Assessment Programme was launched in order to assess teaching activities on degree programmes. This programme, which operated until 2006, promoted the assessment of over six hundred study programmes, 59% of which were assessed by ANECA and the remainder by regional agencies.

The alteration to the juridical framework governing higher education, for the adoption of the Bologna reforms in 2007, had significant implications for the processes of quality assurance, which began to include compulsory assessment of all programmes organised within the framework of the new degree structure157, as regards ex-ante accreditation and ex-post re-accreditation of each programme.

Thus, as part of the responsibilities that are legally conferred upon it, but also considering its task to support institutions in their adaptation to the new processes of accreditation, ANECA has developed a variety of assessment, accreditation and certification processes, namely:

- The VERIFICA Programme for assessment of proposals for the creation of new 1st and 2nd cycle study programmes, in conformity with the national qualifications framework, with a view to ex-ante accreditation. Assessment is carried out according to a handbook which specifies ten criteria and guidelines to be followed by institutions, which focus on around forty appraisal points established by law. The Council of Universities has the final responsibility for the accreditation decision, but this requires a positive appraisal from ANECA.

- The DOCENTIA Programme aiming to provide a framework of reference for appropriate assessment of the teaching activity, taking into account international good practices. This programme was launched in 2007 and aims to support universities in the design and implementation of their own models of assessment of teachers, bearing in mind three basic aspects of teaching competence – planning of teaching, classroom activity, including the means of assessing students, and learning outcomes. Participation in the programme is voluntary and may lead to certification of the model developed by the institution, following positive appraisal by ANECA, which occurs after at least two years of follow-up accompaniment of the procedures established. These procedures should be based on four criteria: fitness for purpose of the teaching undertaken by the teacher, regarding the requirements established by the

156 Currently there are eleven regional Agencies: AGAE (Andaluzia), ACPUA (Aragão), AQUIB (Ilhas Baleares), ACECAU (Canárias), ACUCM (Castilla-la-Mancha), ACSUCYL (Castela e Leão), AQU (Catalunha), ACSUG (Galiza), ACAP (Madrid), UNIQUAL (Basque Cuntry) and AVAP (Valencia). 157 The offer of programmes adapted to the Bologna Process began in the academic year 2008/09 and as from 2010/11, inclusive, Spanish universities will not be allowed to admit new students to the programmes which are not adapted to the new degree structure

78

institution and the learning objectives set out in the curricula; the level of satisfaction with the work of the teacher on the part of students, academic managers and colleagues; the efficacy and capacity of the teacher to develop the desired competencies in the students; innovation in teaching methods158.

- The AUDIT Programme, also launched in 2007 in cooperation with the Catalunian and the Galician Agencies (AQU and ACSUG)159, with the aim of providing guidelines to support institutions in the development and implementation of their systems for internal assurance of quality of teaching and in putting into practice a process leading to recognition of these systems.

- Teacher assessment programmes aimed to guarantee that the applicants for teaching places at the universities, either as state employees or on a contract, respect the minimum scientific and academic requirements to apply for these places in each of the institutions (this corresponds to the creation of a national pool of candidates who are certified to apply for the job positions opened up by the institutions themselves):

• The ACADEMIA Programme of national prior accreditation for access to the university teaching body, to the position of Profesor Titular de Universidad or Catedrático de Universidad, which assesses the profile of potential candidates;

• The PEP Programme of assessment of teachers for the effect of contracting, which assesses the teaching and research activities and academic background of potential applicants to the categories of contracted university teachers, which are present in the law (profesor contratado doutor, profesor ayudante doutor, profesor colaborador and profesor de universidad privada).

- The Quality Mention Programme in Doctoral Studies (PDC), for the recognition of the quality of doctoral programmes and of the structures which support them.

- The Assessment of Services Programme, which provides university services with an instrument for support and reflection, for their assessment, review and action for their continuous enhancement.

As part of its concern for external monitoring of accredited programmes, ANECA is developing a new initiative, the MONITOR programme, in collaboration with the regional agencies, in order to produce handbooks for the monitoring of study programmes in the period between accreditations, on the basis of publically available information160.

Re-accreditation of study programmes has not yet been formally regulamented. Following a request from the Ministry of Education and Science, the Agency drew up a proposal for a model for ex-post assessment of accredited 1st and 2nd cycle programmes, 158 Codina, E. A. & Jiménez, E. G. (2008). Evaluation of teacher competence in Spain: The DOCENTIA programme. In EUA (2008), p. 20. 159 Later, the UNIQUAL Agency (Basque Country) also began to participate in the programme. 160 ANECA (2009b). Plan de Actuación 09, p. 18.

79

in which the criteria and guidelines are established for their re-accreditation161. It is suggested that the process should be applied to each programme, one year after it has become fully functional and, following that, every six years. It appears that no formal decision on the proposal has yet been taken.

The enormous range of assessment programmes mentioned shows ANECA’s broad scope of responsibilities in the area of quality in higher education, but it also reveals the concern for finding a balance between, on one hand, the mandatory intervention of ex-ante and ex-post assessment of programmes for the purposes of accreditation, and, on the other hand, the initiatives to support the universities to prepare for the demands of accreditation processes and at the same time to develop their internal quality culture. A prime example of this facet of the Agency’s work was the launching of the AUDIT and DOCENTIA programmes, in anticipation of the VERIFICA programme. Indeed, among the ten criteria established for course accreditation, there are three (Objectives, Teaching Staff and Quality Assurance System) which are particularly relevant in the context of a change in the model of teaching methods which, it is hoped, will become more centred on students’ learning processes. The DOCENTIA programme has obvious and direct relevance in the area of teaching staff, which has traditionally been a difficult one. In turn, the Audit programme allows participating institutions to prepare for the compulsory existence of an internal system, establishing procedures associated with quality assurance, and also the formal mechanisms for approval, control, regular review and continuous enhancement of study programmes.

As was mentioned earlier, Spanish universities have a long tradition in the field of teaching assessment. For example, surveys carried out among students to establish their degree of satisfaction with teaching have existed since 1981 and were later extended to graduates and employers. The University Quality Plans provided relevant experience for the definition of quality standards in teaching, and the mechanisms installed for the monitoring of these plans provided an opportunity for the development of these procedures and structures geared towards quality enhancement. At the same time, some university schools developed the installation of mechanisms for quality assurance for the processes of conception, review, and improvement of degrees offered, based on the EFQM or the ISO 9000 norms. However, in spite of the variety of such experiments, as a general rule these initiatives did not have a systematic, structured focus geared towards continuous enhancement of the academic offer to students162. 161ANECA (2006). Propuesta – Criterios y Directrices para la Acreditación de Enseñanzas Universitarias Conducentes a Títulos Oficiales Españoles de Grado e Máster. 162 ANECA, AQU & ACSUG (2007a). PROGRAMA AUDIT: Guía para el diseño de Sistemas de Garantía Interna de Calidad de la formación universitaria, p. 7.

80

The design of the AUDIT programme took this into account and defined a methodology for the development of internal quality systems which builds upon the activities and mechanisms for quality assurance which are already in place, which act as starting points for the construction of a coherent and complete quality system. In other words, the programme does not impose a model for internal systems of quality assurance, but rather encourages institutions to work on elements they already have available, providing individually tailored support to each of the participating institutions.

The AUDIT programme was designed to develop in three sequential phases: - The project phase of the internal quality assurance system, which included the

preparation and diffusion of a handbook163 by the Agency, and which implies the definition and documentation of the respective quality systems by the participating institutions;

- The verification phase, with a view to allowing institutions to guarantee the fitness for purpose and the relevance of the project for its internal quality assurance system with regard to established requirements, before moving on to the its implementation;

- The certification phase, which permits certification of the internal quality assurance system by the Agency, after its effective implementation.

The initial phase of design of the system includes four stages to be developed by the institution:

- Formal, explicit, institutional commitment to the development of an internal quality assurance system by the governing bodies of the institution, namely with regard to allocation of the resources necessary for the development of the project and the definition of the organ, structure or group responsible for the system project;

- The planning of the process for the system project, namely concerning the identification of actors, their responsibilities, the planning of the different stages to be developed, the means anticipated for the involvement of the instructional staff and to transmit the basic aspects of the project, and the process of its approval;

- An internal diagnosis of the existing situation, which aims at providing guidelines, in order to obtain a general vision of the organizational structure of the institution, the legal and normative conditions which limit it, and the procedures and mechanisms already in place associated with enhancing the academic offer, which may serve as a point of departure for the quality system project;

- The definition of the internal quality assurance system, in terms of the policies and quality objectives to be achieved and the methods and procedures to be used, taking into account the guidelines of the AUDIT programme and the documentation pertaining to all the components planned for the system.

163 Op. Cit.

81

To guide and facilitate the work of the institutions, two complementary tools were developed: a handbook of guidelines for the design of internal quality assurance systems and the way to document it164, and a detailed set of tools for the diagnosis to be undertaken165.

A first group of 58 universities (around three quarters of the total number) agreed to voluntarily participate in the programme, either as a whole, or through some of its schools. This process involved the preparation of 125 different projects, which reveals the interest generated by this tool.

In the meantime, the participating agencies prepared the verification phase, and in 2008 published a handbook of guidelines for the assessment of projects undertaken by the institutions166, proceeding to their evaluation. The first results showed that only 27% of the projects analysed were considered as ‘positive’, and the remaining 73% were considered ‘limited positive’167, which means they had to remedy the weaknesses identified and undergo a second verification. These results reveal the advantages of the intermediate stage of ex-ante assessment, as it allows the majority of institutions the possibility to identify aspects of the design of their internal quality assurance system which did not totally comply with the norms set out for the process of certification, and enabled them to correct them in the design phase, before they were fully implemented, with obvious gains in terms of time and resources.

As was mentioned previously, guidelines were produced for the diagnosis, design and assessment of internal quality assurance systems, with the double aim of supporting the institutions in the initial diagnosis and in the design of their systems and also in establishing points of reference to be used in the ex-ante assessment process at the verification phase, and in subsequent certification of the system. These guidelines, which closely follow the ESG, may be summarized in the following standards, which question the institution on the way in which it:

- Defines its quality policy and objectives; - Assures the quality of its academic programmes; - Centres its teaching on students; - Assures and enhances the quality of its teaching staff; - Manages and improves its material resources and services;

164 ANECA, AQU & ACSUG (2007b). PROGRAMA AUDIT: Directrices, definición y documentación de Sistemas de Garantía Interna de Calidad de la formación universitaria. 165 ANECA, AQU & ACSUG (2007c). PROGRAMA AUDIT: Herramientas para el Diagnóstico en la implantación de Sistemas de Garantía Interna de Calidad de la formación universitaria. 166 ANECA, AQU & ACSUG (2008). PROGRAMA AUDIT: Guía de Evaluación del diseño de Sistemas de Garantía Interna de Calidad de la formación universitaria. 167 ANECA (2009a). Memoria de Actividades Enero – Diciembro 2008, p. 32.

82

- Analyses and takes into account the outcomes (of learning, of integration into the work market, and satisfaction of the different stakeholders);

- Publicises information on the degrees offered.

Although the standards emphasise the methodology aspect (‘how to’), each of them is divided into a series of guidelines which specify basic features to be included in the system, including those for meta-evaluation, as can be exemplified in the standard regarding quality policy and objectives:

- The institution should consolidate a quality culture supported by a quality policy and quality objectives which are publically known and accessible. Namely, the institution should:

• Make a written, public declaration which expresses its quality policies and objectives;

• Indicate the stakeholders involved in the definition of the quality policy; • Integrate the different components (organs, procedures, processes …) for

the configuration of a system which permits the realization of the said quality policy;

• Set up actions to define, approve, review and improve the quality policy and objectives;

• Determine the means of being accountable (how, who, when) to different stakeholders regarding the achievement of the quality policy and objectives.

The handbook for assessment of the design of internal quality assurance systems, which corresponds to the verification stage, specifies a set of assessment criteria regarding the clarity and coherence of the organization of the system, the compliance with the Agency’s guidelines, the clear identification of responsibilities and the commitment of the institution’s governing bodies regarding the implementation and continuous enhancement of the system, the monitoring and review mechanisms, the performance indicators generated by the system, and the mechanisms for continuing quality enhancement which should be based on realistic, measurable and duly timetabled quality objectives. Based on these criteria, the audit team will assess how far each of the elements complies with the guidelines, expressing its appraisal on a four-level value scale, according to the following criteria:

- Satisfactory (A): There exists documentary evidence that the conditions established in the directive are complied with;

- Sufficient (B): There is documentary evidence that at least the basic aspects of the conditions established in the guidelines are complied with, even though some secondary aspects are not;

- Insufficient (C): There is some documentary evidence of some of the contents related to the guidelines, but this does not comply with all the basic conditions and is not correctly defined or not organized in a systematic way;

- Absence of information (D): The documentary evidence concerning compliance with the directive either does not exist or is unsuitable.

83

For the third phase of the AUDIT, the mechanisms and protocols for accompaniment, assessment, and certification of implemented internal quality assurance systems are presently (in January 2010) still in the preparation phase. Meanwhile, an invitation was extended for the creation of a second group of participating institutions.

In brief, the Spanish national quality assurance system is complex and relatively burdensome, and includes procedures for assessment, accreditation, and certification. Compulsory quality assurance processes include the ex-ante accreditation and reaccreditation of study programmes at all levels of higher education, as well as the previous individual certification of all applicants for university teaching positions, through a process which is conducted at national level. The ANECA, in collaboration with the regional agencies, developed a variety of other programmes with potential impact on the quality of higher education with a view to supporting institutions in the planning, installation and enhancement of their internal quality assurance systems, or of components of these systems. These include the assessment of the teaching activity or the assessment of services, which although of a voluntary nature have seen strong participation on the part of universities, which means that they are considered useful for institutional development.

However, although these programmes are in compliance with the norms and conditions of programme accreditation processes, which will make it easier for institutions certified, for example, by the AUDIT or DOCENTIA programmes to demonstrate their conformity with accreditation standards, to date, no simplification of the compulsory accreditation processes or lightening of their scope is foreseen. Thus, it would appear that a sufficient level of public confidence in the institutions of higher education and in the quality of the degrees they award has not yet been reached, and consequently it is not yet possible to invert the movement of the pendulum to start shifting away from accreditation of study programmes, as has been happening in other countries. There may be a case for asking whether the joint work which is being done between the agencies and the institutions, in the ambit of the multiplicity of programmes of quality assurance, will create in a near future a climate suitable for approaches of a lighter nature in external quality assurance in our neighbouring country. This would be important because, after the current phase of evident enthusiasm, institutions will certainly become weary when it comes to the time of reaccreditation of study programmes, which, if repeated along the same lines, it is generally agreed, bring very little in the way of added value.

84

7.2 CORE CHARACTERISTICS OF THE INSTITUTIONAL AUDIT PROCESSES – A COMPARATIVE ANALYSIS

The case studies presented above reveal a dynamic and diversified situation in the European panorama of external quality assurance, confirming the influence of the national context on the initiatives undertaken by the agencies. For this reason, as was said earlier, a direct comparison of different approaches to the processes of institutional audit is not an easy task, since these processes are generally dictated by the very particular circumstances of national systems of quality assurance, which are fundamental for the understanding of the models adopted or being developed.

There are, however, some trends which are worth analysing. The most noticeable is the progressive compliance with the ESG, with considerable impact on the design of procedures, and which is leading to some convergence of criteria, standards and practices in quality assurance processes, which is reflected in audit processes. A second, equally visible, trend is the continuous evolution and adaptation of external quality assurance processes. Indeed, it can be said that, as a general rule, no particular approach is repeated in a new assessment cycle without significant alterations, and in some cases the changes introduced or planned are substantial. Here, it should be noted that, in the majority of these cases, these changes were not the result of a detailed assessment of the models that were being applied, but were mainly due to alterations in the political agenda and in the balance of power between the partners involved in educational policies.

When considering the evolution of the systems of quality assurance, another feature worthy of analysis is the question of how far the national audits are contributing to simplification of the external quality assurance processes, confirming, or otherwise, the notion that these procedures will tend to become lighter as greater confidence is felt in the internal quality assurance systems of the institutions.

With this in mind, a brief overview follows of the similarities and differences between the national institutional audit systems, in an attempt to identify common components, which are organized around key aspects.

7.2.1 Scope and motivation behind audit processes

Audit processes are not directly concerned with the quality of the activities undertaken by the audited institution, but rather with the way in which the institution assures the quality of the activities it undertakes, through quality management procedures. During the auditing process, it is up to the institution to demonstrate that the

85

objectives it has established (either alone or in collaboration with relevant external partners, namely politic powers) are indeed attained and are continuously monitored and enhanced.

Regarding the quality of teaching and learning, the ESG establish benchmarks for internal systems of quality assurance which, in one way or another, are embedded in the standards adopted by the different auditing systems. All the agencies that develop auditing processes are particularly concerned with teaching and learning, but in the case of other thematic areas, the agencies use diversified approaches. This is the case of research, which in many countries is assessed by a different organ, but which in others may be part of the institutional auditing process, as happens, for example, in Austria. The agencies’ approaches are also varied when it comes to the way in which evidence on the efficacy of the internal systems of quality assurance is analysed, ranging from random samples of processes and study programmes in Germany, to ‘audit trails’168 in England and Northern Ireland, or the examination of key processes in specific performance areas in Austria.

As a general rule, institutional audit processes are presented as lighter and less intrusive forms of external assessment, when compared to evaluation and/or accreditation of each individual study programme offered. They also claim they intend to contribute to making each and every institution responsible for assuring quality and that through this margin of manoeuvre they create space for institutions to use creative approaches, geared towards systematic quality enhancement. This being so, the criteria established for assessment by the auditors tend not to be prescriptive in relation to the way in which the institution should organise its quality assurance system. As a result, emphasis is placed not on the internal quality procedures themselves, but on their respective efficacy in terms of their capacity to identify cases where quality is lacking and to promote improvement.

However, the reasons that led to the adoption of institutional auditing process are varied, which is reflected in the aims of the audits. From a comparison of the different systems analysed, it can be concluded that:

- In cases in which institutional audit is compulsory, the function of control over the institution, for the purpose of the maintenance of public recognition/ accreditation of the institution and/or access to public funding, would appear to

168 Normally, during each audit, two audit trails are undertaken, which correspond to the scrutiny by vertical sample through different levels of management, focusing on the implementation and results of particular institutional policies and procedures, previously agreed upon with the institution, with a view to a detailed analysis of the way the policies and procedures are effectively put into practice (QAA, 2009, p. 11).

86

dominate169, which does not exclude, however, the contribution the process may make to a better internal organization of quality assurance procedures, resulting from the information and benchmarks provided by the Agency, the contacts with specialists from the Agency and the audit team, as well as the recommendations made in the reports. Indeed, institutional audit processes always aim, at least implicitly, to encourage and support institutions in promoting quality assurance and enhancement, though there is the danger that, when faced with the possible penalties resulting from potential shortcomings detected during the compulsory audit, institutions will focus on formal aspects of compliance with standards established by the agencies and will attempt to embellish self-assessment reports. This would be to the detriment of a reflective, serious analysis and of the implantation of procedures and attitudes which are truly conducive to excellence in the accomplishment of their institutional mission.

- When participation in audits is voluntary, the predominant aim tends to be quality enhancement, although the audit process does include verification of compliance with some external references, in particular with regard to the ESG. The fact that the voluntary audit processes are being received favourably by higher education institutions indicates that they consider them to be an important added value. There are two determining factors which contribute to the high uptake of audit processes by institutions: on one hand, the obligation for institutions to establish internal systems of quality assurance places them under strong external pressure and it is therefore natural that the specialised information and support given throughout the audit process is seen as a useful tool to help them to fulfill this obligation; on the other hand, certification of quality systems represents a seal of quality, which the institution uses in the presentation and marketing of its study programmes and services.

- In the case of Germany, and probably also in the Netherlands and in Flanders in the next assessment cycle, there is an additional incentive, insofar as higher education institutions are given the choice between the existing regime of cyclical accreditation of all the study programmes or the new institutional audit process which will simplify the accreditation of programmes (in Germany there will be sample accreditation of around 15% of programmes and in the Netherlands and Flanders the process of accreditation is expected to become less burdensome, though it will be applied to all the programmes).

In another, complementary, aspect of the analysis, regarding the scope of the institutional audit processes, it is clear that in the ten cases analysed:

- In four of them, the audit process is compulsory and is the only, or principal, method for external quality assurance, associated to the awarding of institutional accreditation (the case of Norway) or access to public funding (the situation in England and Northern Ireland, Scotland and, in part, in Switzerland);

- In three countries, the process is voluntary and complementary to the individual accreditation of study programmes:

169 Scotland is an exception, given that, although the ELIR is obligatory, it is clearly geared towards quality promotion.

87

• In Germany and in upcoming cycles in the Netherlands and in Flanders, the audit process is an alternative to accreditation of study programmes;

• In Spain audit is cumulative with the accreditation of study programmes, and is designed as a seal of quality and a process of support for the development of the institution;

- In three other countries, the audit process is both voluntary and the only way of external quality assurance. It is strongly focussed on the encouragement of quality enhancement and is essentially seen as a seal of quality (Finland and Austria) and as support for institutional development (Finland, Austria and until recently, Denmark);

- In all of them, with the apparent exception of Spain, it tends to represent a lighter means of external quality assurance than the kind in use in countries where study programme accreditation predominates.

7.2.2 Criteria and methodology

A study of the handbooks for institutional audit produced by the different agencies shows that the methodologies adopted have some common traits, but also marked differences. In all cases, as would be expected, there is prior definition of the criteria to be used in the audits. These are frequently expressed in the form of reference standards and specific criteria for the assessment of the respective level of attainment or observance. The standards, in turn, are normally formulated as declarations which characterize a well-established and mature system for the assurance and promotion of quality. In general, the standards are not prescriptive regarding the actual contents of internal quality assurance systems, but rather contain general guidelines for their development. As for specific assessment criteria, there are differences in the demand for conformity of the internal quality assurance system with the standards, ranging from the explicit declaration from NOKUT (Norway) that full compliance with each of the ten standards established by the Agency is not required, to the need for full compliance of the seven minimum requirements established by the OAQ (Switzerland).

A common characteristic is the degree of conformity with the ESG of the standards established by the agencies. They all indicate the ESG as an essential external reference, which in some cases has led to recent adjustments in a move towards improved conformity with the Europeans standards, and there are many cases in which the formulation of the standards themselves follows the ESG text very closely. There are, however, other important external references, as is the exemplary case of the Academic Infrastructure170 in England and Northern Ireland and in Scotland, and also the references to good practices gathered from audits, namely through the thematic analysis

170 See 7.1.1.

88

of reports, and also references arising from discussion of National Themes as in Scotland.

Regarding the way the audits are conducted, the most usual practice is the four-phase model which is common to processes of external quality assurance (self- assessment report, visits, audit report, and publication of the report), with some nuances. Some agencies emphasise the simplified nature of the self-assessment report itself insofar as the audit should concentrate on documentation of the description of the system and the demonstration of the efficacy of its operation which should already exist as part of the design of the system. In Norway, for example, a self-assessment report is not required, and only normal documentation of the quality assurance system and annual reports of activities are requested, under the assumption that these reports will include data providing evidence of the operation of the system.

As a rule, the agencies stress the vital role of dialogue and interaction with higher education institutions for the development of models of institutional audit, with a view to adapting them better to the perceived needs of the institutions and, at the same time, to the acceptance of the model and to full commitment to it on the part of the institutions. In a brief overview, the following observations, emerging from the case studies presented, and which would appear to be reiterated in the results of the survey undertaken within the informal network which was mentioned in section 6.3171, are wort noting:

- England and Northern Ireland: The external reference constituted by the Academic Infrastructure was developed in close cooperation with the QAA, the institutions and other stakeholders and agreed upon with institutions.

- Scotland: The establishment of the Quality Enhancement Framework, which integrates the institutional assessment programme ELIR, corresponded to the desire of the higher education institutions for the development of approaches to a review process more geared towards the promotion of enhancement. The effective partnership established between the Agency, the body responsible for funding (the Scottish Funding Council), the Rectors’ Conference, the individual universities and the student organizations is considered to be an important element within the system.

- Norway: The alterations to the juridical framework leading to the creation of NOKUT and the setting up of the institutional audit system were part of a wider ranging reform of quality in higher education. It benefited from a long tradition of confidence enjoyed between the institutions and the authorities. The audit criteria were adopted after consultation with the higher education sector.

171 AQA (2009b). Trends of Quality Assurance and Quality Management in Higher Education Systems.

89

- Switzerland: The universities were involved in the design of the audit model and had a vital part in the development of the standards through a ‘quality network’.

- Finland: A great effort was made to create a climate of confidence in the audit process, and constant dialogue was established with the institutions. The audit model and criteria were widely discussed with institutions, which unequivocally accepted them.

- Denmark: The audits, which were carried out on a completely voluntary basis, were widely accepted by the universities and there was particular appreciation for the dialogue involved. The decision to suspend the institutional audit process and to substitute it with accreditation of study programmes was strictly political in nature.

- Germany: The German Rectors’ Conference participated in the process which led to the adoption of the model of system accreditation. In the audits undertaken by AQUIN there is a preliminary conversation with the institutions to decide on accreditation objectives and a formal contract is established to set out the scope, the timing and the costs of the process. In its formal response to the panel’s report, the institution is invited not only to point out factual corrections, but also to present comments and to describe methods for improvement initiated in the meantime.

- Austria: The audit process was set in motion with public universities in mind, at a time when they were involved in the search for consensual ways to demonstrate the quality and the efficacy of their quality assurance mechanisms. The universities were consulted in the development of the model and appear to be showing considerable interest in the process. The Support and Consultancy function of this model has strengthened dialogue with the institutions.

- The Netherlands: The audit model, which is in the pilot study phase, is the result of an agreement between the Agency, the government and higher education institutions. The close involvement of the stakeholders was considered fruitful and worthwhile, in particular because it led to the commitment of the whole sector to the new model172.

- Spain: The way in which the Audit programme is conceived, with an intermediate checking phase, naturally encourages dialogue with the institutions.

Other procedural aspects considered important are, for example, the need for proper training of auditors, the advantage to be gained from a formal contract with the institutions for the performance of the audit, and previous meetings before the audit is begun, for example in the form of a seminar which is informative in nature. Some agencies emphasise support to institutions in the development of their own internal quality systems, namely through the provision of methodological guidelines for the conception of their systems, such as in Spain, or a support and consultancy service, which is available in Austria.

172 van Galen, S. & Woutersen, M. (2009). Balancing Quality Enhancement and Accountability – Reforming the Dutch and Flemish Accreditation System, p. 6.

90

7.2.3 Consequences of audit processes

As was seen earlier, there are variations in the aims and objectives of institutional audit, associated with the global framework of higher education in each country. In countries in which the audit processes are compulsory, there is always a formal decision at the end of the procedure which is expressed in terms of approved or not approved, or an appraisal of confidence in the quality system of the institution, graded into total confidence, limited confidence or no confidence. A similar situation exists in countries in which institutions may opt for either institutional audit or accreditation of study programmes173. A favourable result in the audit has effects at two different levels:

- Confirmation of institutional accreditation and/or access to public funding; - The strengthening of the image and prestige of the institution through the

awarding of a seal of quality and the corresponding impact on public opinion, echoed in the media.

In voluntary audit processes, the result is also expressed as a formal decision, with the exception of the regime operating in Denmark. Approval in the audit leads to a certificate of quality and has the same effects regarding prestige as mentioned before. This effect is far from negligible and in practice constitutes a way of putting pressure on the institutions to participate in the audit schemes, as was seen in England, for example, where strictly private institutions which do not apply for public funding voluntarily take part in the audit process174. Perhaps for this reason, it is noticed that agencies with experience in this field are unanimous in affirming that acceptance of auditing is widespread and on the increase175.

In cases in which the audit process is compulsory, the possible consequences of an unfavourable result would, in principle, be drastic, but in reality all systems tend to adopt, in first instance, a process of timely rectification in all areas found to be lacking. In practice, this means that the foreseen sanctions are rarely applied, as may be seen from the following summary of the types of measures taken when an unfavourable result occurs:

- In England and Northern Ireland the institution is given three months to present a plan of action aimed to put into practice the ‘essential recommendations’ and ‘advisable recommendations’ which appear in the audit report, to be accomplished within an eighteen-months period, with three-monthly progress reports. If doubts still remain at the end of this period, the date of the next audit

173 In the case of Germany, for reasons of a juridical nature, the notion of accreditation is maintained, so the decision is formally expressed in terms of accreditation/no accreditation of the system. 174 Brink, C. & Kohler, A. (2009). Audit Approaches – Features for a Common Understanding. In AQA (2009b), p. 43. 175 Ibidem, p. 42.

91

is brought forward. No cases of loss of funding have yet occurred as the result of auditing.

- In Scotland the institution is given the chance to correct the aspects that led to the unfavourable result, by submitting an action plan, the execution of which is monitored very closely and which normally involves a follow-up visit. The actual methods to be used and the timing of the follow-up are agreed upon case by case by the Agency and the Funding Council, according to the nature of the action plan, with the institution also being heard. The repetition of an unfavourable result would bring serious consequences. In the audit cycle which finished in 2007, there was only one case of limited confidence regarding the issue of academic standards, which was one of the key reasons for the fusion of the institution with another university.

- In Norway, the institution is given a six-month period to rectify shortcomings, at the end of which a new audit is carried out. If the institution fails again, it loses the right to create more study programmes and to admit new students, but it may request a new audit one year later. Between 2003 and 2008, 54 institutions were audited and negative results occurred in eight of them, four of which were approved in the follow-up audit.

- In Switzerland there are no immediate effects, but a year later the institution is audited again. If the result of this second audit is negative, the institution faces a cut in federal funding of up to 25%, a situation which is considered highly unlikely by the OAQ Agency176.

- In Germany and in the system foreseen in the future for the Netherlands and Flanders, the institution does not acquire the status that would allow its access to the simplified accreditation of its courses, and is required to undergo the normal process of individual accreditation.

The situation is different when the audit process is totally voluntary, insofar as the main consequence of an unfavourable result is not being awarded the seal of certification. There may also be negative effects on the image of the institution if the result is made public, and potentially in the negotiation with the ministry regarding the contract for performance objectives (in the case of Austria and Finland). As the emphasis of the exercise is on enhancement, the report of an unfavourable audit will also contain useful recommendations for rectifying the shortcomings which led to a conditional approval, or non-approval, thus facilitating the rectification process and the preparation for a new audit. For example, the Finnish agency FINHEEC claims that for the 19 audits undertaken, four led to the need for a new audit, and that, although there were no consequences in terms of funding, the institutions took these results very seriously and made preparations for the new audit two years later177.

176 Ibidem, p. 43. 177 The reasons given for the negative decision have to do with the lack of scope and efficacy of the quality assurance system (Moitus & Seppälä, 2009, p. 71).

92

7.2.4 Follow-up

All of the agencies analysed established follow-up mechanisms, of one sort or another, for institutional audits, in some cases on a voluntary basis and in others with formal procedures, which may include, for example, a compulsory assessment in the middle of the cycle. In concrete terms, the situation is as follows178:

- In Switzerland, Norway, Austria and Denmark, no formal follow-up procedures are established, apart from the requirements regarding re-audit in the case of an unfavourable or conditional outcome. There is, however, reference to the possibility of follow-up conferences or seminars, when requested by the institution. It is also mentioned that the cyclical nature of the audit process is in itself a type of follow-up179.

- In Spain, the methodology for the AUDIT programme is being developed stage by stage, so the frequency of audits and the follow-up mechanisms are still under discussion. In the Netherlands, the situation is similar, as the process is still at a pilot stage.

- In England and Northern Ireland, follow-up occurs in the middle of the cycle and consists of the submission of a brief report by the institution and its analysis by two senior members of the QAA Agency.

- In Scotland, annual discussions are held between the Agency and each institution, with the participation of one member of the Agency and a small group of representatives from the institution, in order to discuss developments that have occurred during the year. This method of annual meetings puts the focus on dialogue, and there is no preparation of a report by the institution and no formal appraisal by the Agency. However, a year following the publication of the audit report, the institution is required to produce a follow-up document regarding the measures taken as a result of the audit and their level of efficacy, which will be put to discussion in the following annual meeting.

- In Finland, follow-up and benchmarking seminars may occur on a voluntary basis, three years after the audit.

- In Germany, the institution should arrange external assessment of a sample of study programmes four years after the audit (only for feedback purposes, and involving no formal accreditation decision).

7.2.5 Summary table

In order to provide a rapid, general outline of the current situation in each of the countries studied, the tables on the following pages contain brief comparative summaries of the nature and main characteristics of the respective institutional audit systems (Table I) and of aspects related to their organisation (Table II).

178 Brink, C. & Kohler, A. (2009). Audit Approaches – Features for a Common Understanding. In AQA (2009b), p. 55. 179 Indeed, it should be accepted as a basic principle that, in the exercise of self-evaluation for the purpose of external audit, specific mention should be made of the ways in which considerations and recommendations of previous evaluations were dealt with.

Table I – Comparative summary of type and main characteristics of national institutional audit systems in Europe

(a) – Audit process predominantly for control or more geared towards continuous enhancement of quality. (b) – Process introduced in 2007, but not yet concluded. (c) – In future cycles it will be 8 yearly. (d) – Non-accredited institutions lose the right to .apply for accreditation of new programmes.

Consequences of a negative decision Country Type (a) Obligatory or Voluntary Year of commencement Frequency

Formal Decision (certification of the

internal quality system) On funding On accreditation/ others

England/Northern Ireland

Control (in transition to enhancement )

Obligatory (for the purpose of funding)

1990 (current model in

2006) 5 to 6 years

Formal decision: • Confidence • Limited confidence • No confidence

Yes (No cases have yet

occurred to date)

Execution of corrective action plan, within 18 months maximum.

Scotland Enhancement Obligatory 2003 4 years • Confidence • Limited confidence • No confidence

Yes (only if audit failed

after corrective action plan)

Submission of corrective action plan, followed by another visit. Close monitoring until decision of confidence is attained.

Norway Control and enhancement in combination Obligatory 2003 no more

than 6 years • Approved • Not approved

No

Has 6 months to correct shortcomings. If this fails, loses institutional accreditation. May trigger general supervision and revision of accreditation (d)

Switzerland Enhancement / Control (attempts to balance the

two) Obligatory 2003 4 years • Pass

• Fail

Yes (only if fails the repeated audit)

Re-audit one year later. Failure in this audit leads to cuts of up to 25% in federal funding.

Denmark Enhancement Obligatory (by agreement with the Rectors’ Conference)

2003 (suspended in 2008) No formal decision No No formal consequences.

Finland Enhancement Voluntary

(100% participation in 1st audit cycle)

2005 6 years • Certified • Re-audit

No

New audit, focusing on corrective measures taken. May be discussed in contractualisation with the Ministry.

Germany Control

(with some elements of enhancement)

Voluntary An alternative to the individual

accreditation of study programmes)

2008 6 years (c) • Accredited • Not accredited

No

Must submit all study programmes to a normal process of accreditation

Austria Enhancement Voluntary 2008 6 years • Pass • Conditional pass • Fail

No

No certificate issued. May have effects on contractualisation with the Ministry.

Netherlands Control

Voluntary Will allow simplified accreditation of study

programmes

2008 (in the pilot phase) 6 years

Attribution of status of: • Earned trust • Limited trust • No trust

No

Does not obtain the statute of ‘earned trust’. Must submit all study programmes to the normal process of accreditation.

Spain Enhancement Voluntary

(over 80% of universities participated)

2007 (b) Perhaps 3 to 4 years

• Pass • Conditional pass • Fail

No (but planned for the

future)

No formal consequences (but may influence the results of the study programme accreditation process) Published on the Agency’sweb page

94

Table II – Comparative summary of organizational aspects of national systems of institutional audit in Europe.

Country

Scope of the audit

Criteria/standards

Duration of visits - days

Follow-up

England/Northern Ireland

Capacity of institution to manage standards for qualifications and learning quality (with emphasis on students and on learning )

Criteria for a verdict of Confidence, Limited Confidence , No confidence, based on the Academic Infrasrusture.

1 (prelim.) 3 (1st) 4 (2nd)

Follow-up report in the middle of the cycle.

Scotland

Capability of internal quality systems to monitor and maintain quality. Approach promoting effective learning experiences. Strategic approach for quality enhancement.

General guidelines for the audit based on the Academic. Infrastructure and other external references (for example, results of the National Themes).

2 (1st) 3 to 5 (2nd)

Annual meetings with a member of the Agency. Follow-up report at the end of the first year.

Norway (2nd cycle)

Documentation and results of internal quality system (involves all aspects of educational quality).

A set of five criteria, focusing on the quality of the system and the HEI’s use of and benefit from it, closely following the ESG (total conformity is not demanded).

1 (1st) 2 to 3 (2nd)

Action plan and reaudit, in the case of negative decision. Annual seminar for evaluated HEIs.

Switzerland Implementation of internal quality assurance system at the level of teaching.

Seven standards establishing minimum compulsory requirements for the system, in total accordance with the ESG.

2,5 to 3 No systematic follow-up.

Denmark Area of focus agreed upon with institution (teaching, research, innovation ...).

Twelve criteria correlated with the ESG. 2 to 4

No formal follow-up. A conference is held if the institution so wishes.

Finland Global internal quality assurance system. Management of quality in teaching, R&D, interaction with society, support services and management of human resources.

Seven standards defined as target areas for the audit. Analysis criteria which specify four stages of development for each target area. 2 to 5

Compulsory follow-up when a re-audit is requested. Voluntary in the middle of the cycle, with the organization of a seminar.

Germany Results, documentation and formal procedures of the internal system of quality management and assurance.

Generic criteria organised in six standards, closely following the ESG. 1 (1st)

4 (2nd)

Evaluation of a sample of study programmes in the middle of the cycle (no accreditation decision involved).

Austria System of quality management and assurance in performance areas (teaching, research, human resources, internationalization).

Six standards, in line with the ESG. Includes guidelines and exemplification for each performance area. 1 (1st)

2 to 4 (2nd)

No formal follow-up. It is considered “part of the concept”

Netherlands Internal system of quality assurance at the level of teaching (complemented by specific accreditation of study programmes).

Five standards for the system, related with the ESG. Three specific standards for simplified accreditation of study programmes.

(Process still in pilot phase)

Spain Internal quality assurance system in teaching/learning (in the areas defined by the ESG).

Seven guidelines for the design of the system, closely following the ESG. Analysis criteria with classification scale of four categories.

2-3 (probably)

Under discussion.

95

8. CONCLUSIONS

8.1 SPECIFICATION OF INTERNAL QUALITY ASSURANCE SYSTEMS

Internal quality assurance systems in higher education are currently compulsory in all Bologna Declaration signatory states, in line with the basic principle that quality and quality assurance are the prime responsibility of the higher education institutions themselves. The process has been compulsory for many years in some countries but, without doubt, the adoption of European standards in Bergen in 2005 and their transposition into the national juridical frameworks made a decisive contribution to the current prominence of this issue.

However, as a rule there are no specific details of the way in which these internal systems of quality assurance should be organized and should function. It is, therefore, up to each institution to define and implement their own quality system according to their institutional mission, objectives and culture, following the current national guidelines established by law, or the guidelines of the agencies, which also include the ESG guidelines.

In countries in which institutional audit processes focusing on the fitness for purpose and efficacy of internal quality systems are being instituted, the standards established by the agencies tend to be non-prescriptive, encouraging individual and innovative approaches, oriented towards the development of an institutional quality culture more centred on the promotion and continuous enhancement of quality than on the strict, bureaucratic conformity with external standards. In some of these countries, (with emphasis on Scotland and on Norway), the question of quality assurance – external and internal – was even dealt within the wider scope of a reform of quality in higher education.

Without detriment to the freedom of institutions to define their internal quality system, a considerable effort has been invested by agencies, in articulation and dialogue with the institutions and other stakeholders, in the preparation and adoption of guidelines which support higher education institutions in fulfilling this complex task. For example, the Swiss University Conference, together with the OAQ Agency, published guidelines to that effect to be adopted by Swiss universities, and in Spain, the ANECA Agency prepared a handbook for the design of internal systems of quality assurance, complemented by two other handbooks regarding standards to be observed and diagnostic tools. In general, the handbooks for institutional audit include important guiding principles to map out, without unduly influencing it, the work of design and

96

development of the internal quality systems. By associating a set of guidelines to each of the seven standards of internal quality assurance, the ESG themselves give the institutions some valuable suggestions.

By looking at the examples and references analyzed in this study, it is possible to identify some essential specifications for an internal system of quality assurance minimally suited to what can be considered international good practices in this area. A first contribution could be the notion that, underlying the concept of quality assurance, six basic questions may be identified, which have, in fact, been key reference points for institutional audits in the United Kingdom since the 90’s180 and which have influenced other quality assurance systems. These points concern the aspects mentioned below and are usually presented in an interrogative way:

The purpose: What are we trying to do? - The reason: Why are we doing it? - The method: How are we doing it? - Achieving the purpose: How do we know it works? - The optimisation: Why is that the best way of doing it? - Improvement: How can we improve it?

The clear identification of a purpose is, indeed, essential, as is made clear in point 6.2, not only because it may form the basis of the definition of a ‘sense of direction’, which is motivating and provides an impetus for the whole institution, but also because as the quality assurance procedures should be adapted to the purpose, this becomes a central reference for planning and implementation of the system.

In the document which establishes the criteria for institutional audit, the Norwegian Agency NOKUT has an interesting approach to the essential characteristics to be included in a ‘satisfactory system’ of internal quality assurance and which are subject to scrutiny in the audit181. These characteristics, which are formulated in a very concise and open way, could be used as a basic reference. They are:

- The system should be well integrated and closely articulated with the governing mechanisms (steering and management) of the institution;

- The system should generate the information necessary for the accomplishment of its objectives and ensure that the information is analysed and disseminated to the suitable levels of responsibility and management;

- Mechanisms should exist for the utilization of the knowledge obtained through the system in measures aimed at the development and improvement of the institution;

180 Peter Williams, Executive Director of the QAA, claims this approach was first thought of by him in November 1990 (AQA, 2009b, p. 18). 181Criteria for Evaluation of Universities and University Colleges’ Quality Systems for Educational Activities, NOKUT, p. 1.

97

- As is the case with other management tools, the system should be assessed internally and should develop according to needs (as they arise).

It should be added that a good quality assurance system is both a management tool for the institution and a practical system for improvement of its day-to-day activities, so that quality work should not be reduced to inspection rules and control. The system should, therefore:

- Be based on procedures which are intimately associated to the learning processes themselves and to learning contexts;

- Be a source of motivation for all the staff and students and involve them in the work related to quality.

With these basic ideas as a starting point, and taking into account the (incontrovertible) references which are the ESG and the national juridical regime for assessment of higher education182, as well as the reflections presented in the case studies dealt with in this work, it is possible to identify some general orientations for the planning and development of a an internal quality assurance system, in terms of a frame of reference for the organization of the system and some conditions to be fulfilled. The ESG, as well as several of the systems studied, have a scope which is limited to the dimension of teaching and learning, but the Portuguese juridical regime is inclusive in relation to different facets of the mission of the institution. Consequently, out of the ten standards suggested below, seven of them follow the ESG closely and are essentially focused on teaching and learning, and the three others are specifically concerned with research and development, interaction with society and the transversal area of internationalization183. For each of these standards, some general guidelines are suggested which may help to understand their scope and which exemplify their application.

A final note on internal quality assurance systems concerns the nature of the structures for coordination and support of the system, relating to the profile of the ‘quality officers’ involved. Indeed, care must be taken due to the risk of quality assurance processes being considered by the academic community as basically a bureaucratic exercise and a waste of time. In a case study of an English university which looks into the role and effect of the central support structures in the development of quality systems, the following points are made, which are of great general interest:

“Those responsible for work associated with quality should not be seen as part of the management concerned only with getting good results. Instead they should be seen as colleagues who work with academic staff and are accepted by them. They

182 Law nº 38/2007, 16th August. 183 Internationalization can also be dealt with in an integral way as part of the other standards. However, if it is dealt with autonomously, it is given a more holistic view and it gains more visibility.

98

must be seen to have a good knowledge and understanding of academic work and of the nature of learning and teaching. (…) Whilst it is important that staff in quality units understand the demands of quality assurance, they must also be able to ensure that the quality assurance processes do not hinder but in fact assist an institution’s concern to enhance its provision and further its mission”184.

8.1.1 Proposal for a frame of reference for internal quality assurance systems

There follows a frame of reference, formulated in terms of statements which characterise a sound and well developed internal quality assurance system, consonant with the ESG and the applicable legal requirements. The idea is to provide a set of general orientations that can support the higher education institutions in the work of designing and implementing their quality systems.

Reference 1 – Definition of a quality policy and objectives: The institution has consolidated a quality culture supported by a quality policy and by quality objectives, which are formally defined and publically available

For the effect, the institution prepared, formally approved and made public documentation which expresses the institutional policy and objectives for quality, which includes, namely:

- The institutional strategy for quality assurance and quality standards; - The organisation of the quality assurance system, indicating the responsibilities

of the different organs and levels of management in quality assurance; - The ways in which students and other (internal and external) stakeholders are

involved in quality assurance processes; - The means of implementation, monitoring and review of the quality policy.

Reference 2 –Definition and quality assurance of educational offer: The institution has in place mechanisms for the assessment and renewal of its educational offer, developing methods for the approval, monitoring and periodic review of its study programmes and degrees.

With this in mind, the institution promoted the definition of: - Procedures and criteria to organise, inform and decide about the processes of

creation, modification, suspension or extinction of study programmes (whether or not they lead to a degree), with the identification of the organs and internal and external stakeholders involved in the process185;

184 Hodgson, K. (2009). Developments to Encourage Quality Enhancement: A Case Study. In EUA (2009b), p. 20. 185 In the terms of article 18, paragraph a), of Law nº 38/2007, these procedures must necessarily include: - The participation of the pedagogic councils and the appraisal of students, namely through those councils

and student associations; - The participation of the research centers which cooperate in the organization and operation of the study

cycles;

99

- Systems for collection and analysis of information, including feedback from the alumni, employers and other external stakeholders, which may be the basis for decisions regarding the continuation, updating or renewal of the educational offer;

- Procedures for regular periodic revision of study programmes (with the participation of external specialists) and to assure the implementation of the improvements defined in the the review process;

- Explicit definition of expected learning outcomes for each programme, which are publically available.

Reference 3 – Quality assurance of learning and of student support: The institution adopted procedures enabling it to promote and assure the quality of the teaching it undertakes and to ensure that it assumes as a fundamental objective to favour the learning opportunities for student.

For the achievement of this objective, the institution: - Pays meticulous attention to the design and contents of each study programme,

and its respective syllabus, providing an explicit, detailed definition of the expected learning outcomes of each subject, as well as of the core concepts to be acquired, resources available, methods of assessment and planning of the teaching activities, with particular attention to the work load of the student;

- Defines the norms and regulations regarding the organisation of teaching and the students;

- Defines procedures for the selection and recruitment of students; - Develops mechanisms to promote social and psychological support of students,

as well as actions promoting integration and academic success, and periodically assesses these mechanisms;

- Promotes research and innovation activities for students; - Defines procedures for the monitoring, assessment and enhancement of

teaching and learning processes and outcomes, ensuring the involvement of students, teachers and other relevant stakeholders;

- Ensures that the assessment of students is undertaken according to previously defined and publicised criteria, regulations and procedures, which are applied in a consistent fashion186;

- The participation of external advisory entities which cooperate with the institution. 186 The ENQA guidelines give great emphasis to student assessment, which is considered one of the most

important components of teaching. According to these guidelines, procedures for student assessment are expected to:

-“be designed to measure the achievement of the intended learning outcomes and other programme objectives; - be appropriate for their purpose, whether diagnostic, formative or summative; - have clear and published criteria for marking; - be undertaken by people who understand the role of assessment in the progression of students towards the

achievement of the knowledge and skills associated with their intended qualification; - where possible, not rely on the judgements of single examiners; - take account of all the possible consequences of examination regulations; - have clear regulations covering student absence, illness and other mitigating circumstances; - ensure that assessments are conducted securely in accordance with the institution’s stated procedures; - be subject to administrative verification to ensure the accuracy of the procedures.

100

- Adopts procedures to assess the professional integration and evolution of graduates;

- Defines mechanisms to deal with complaints and suggestions.

Reference 4 – Research and development: The institution adopts mechanisms to promote, assess and enhance the scientific, technological and artistic activity appropriate to its institutional mission.

The research and development policies of the institution include, namely: - Mechanisms for institutionalisation and management of research187; - Mechanisms for articulation between teaching and research, namely with

regard to student contact with research and innovation activities from the first years;

- Mechanisms for economic valorisation of knowledge; - Procedures for the monitoring, assessment and enhancement of human and

material resources allocated to research and development, of scientific, technological and artistic production, of the results of the valorisation of knowledge and of the results of articulation between teaching and research.

Reference 5 – External relations: The institution adopts mechanisms to promote, assess and enhance collaboration with other institutions and with the community, namely regarding its contribution to regional and national development.

In the ambit of its external relations policy, the institution has in place procedures to promote, monitor, assess and enhance interface and external action activities, namely with regard to:

- Inter-institutional collaboration; - Services to the community; - Cultural, sporting and artistic external action; - Integration in national projects and partnerships; - Its contribution to regional and national development, in accordance with its

institutional mission; - The collection of own income, through the activities undertaken.

Reference 6 – Human resources: The institution possesses mechanisms to ensure that the recruitment, management and training of its teaching and support staff is undertaken with the guarantee of the necessary qualification and competence, in order that they may properly perform their functions

Consequently, the institution: In addition, students should be clearly informed about the assessment strategy being used for their programme, what

examinations or other assessment methods they will be subject to, what will be expected of them, and the criteria that will be applied to the assessment of their performance.”

187 Procedures and criteria for the creation, extinction and management of research units and interface units, generation of funding, incentives to scientific production, ...

101

- Adopted procedures that enable it to gather and analyse information on the need for teaching and support staff (namely job profile, and profile of required competences), according to its human resources policy;

- Adopted mechanisms for collection and analysis of information relating to competences and results of the performance of the teaching and non-teaching staff, with a view to performance assessment, staff development and promotion and recognition of merit, and adopted procedures to regulate and guarantee the corresponding decision-taking, implementation and follow-up procedures188.

Reference 7 – Material resources and services: The institution adopts mechanisms which enable it to plan, manage and enhance services and material resources with a view to appropriate development of student learning and other scientific and pedagogic activities.

For this effect, the institution has in place: - Mechanisms for collection and analysis of information relating to maintenance,

management and suitability of material resources and services189, including students support services;

- Procedures to regulate and guarantee the corresponding decision-taking, implementation and follow-up procedures.

Reference 8 – Information systems: The institution adopted mechanisms which allow it to guarantee the collection, analysis and use of the results190and of other relevant information for the effective management of the study programmes and other activities.

For the effect, the institution: - Adopted mechanisms to obtain information on the needs and expectations of

different stakeholders, in relation to the quality of the educational offer and the services rendered;

- Developed data collection systems regarding results and other relevant data and indicators191, which cover192: • student progression and success rates; • employability of graduates; • students’ satisfaction with their programmes; • effectiveness of teachers;

188 The procedures adopted to ensure that the teaching staff have the necessary qualifications and competences to perform their function should be made available to those responsible for the external evaluation, in order to be assessed in the evaluation reports (Standard 1.4 of the ESG and art. 18º, paragraph b), of Law nº 38/2007). 189 Facilities, teaching equipment, libraries, IT resources, scientific equipment, canteens, etc, including aspects relating to safety and the environment, as well as the specific needs of students with disabilities. 190 Learning outcomes, integration in the work market, satisfaction of stakeholders,etc. 191 The information system should deal both with quantitative and qualitative data, and should include performance indicators. It is important to define a set of minimum indicators at national level which, in terms of time series, may become A Charter of Institutional Progress for the institutions. 192 Cf. guidelines of standard 1.6 of the ESG.

102

• profile of the student population; • learning resources available and their costs; • the institution’s own key performance indicators;

- Defined procedures to regulate and guarantee the processes of decision making, related to the utilization of results, as well as action strategies for improvement of processes and results;

- Identified ways to involve stakeholders in the appreciation, analysis and improvement of results.

Reference 9 – Public Information: The institution adopted mechanisms which permit periodic publication of up-to-date, impartial and objective information, both quantitative and qualitative, concerning study programmes, degrees and diplomas offered and other activities undertaken193.

For this effect, the institution defined procedures for regular provision of public information regarding a pre-defined set of data and results194.

Reference 10 – Internationalisation: The institution adopted mechanisms which allow it to promote, assess and improve its international cooperation activities.

In the ambit of its internationalization policies, the institution defined procedures to promote, monitor, assess and improve activities of an international nature, namely those in relation to:

- Participation/coordination of international education and training activities; - Participation/coordination in international research projects; - The mobility of students, teachers and other staff.

193 Including monitoring their graduates, in terms of employability, for a reasonable period of time (art. 18º, paragraph e) ii), of Law nº 38/2007). 194 Both the RJIES (art. 162º, nº2), and the ESG (in guidelines for the standard 1.7), specify aspects in relation to which the institution should make available precise and sufficient information. From these guidelines and from reflections made during this work, one can conclude that the information to be published should include: the mission and objectives of the institution, and its statutes, regulations and organic units; its educational offer; the learning outcomes and qualifications awarded, and employment perspectives, in relation to each study programme; qualifications of teaching staff and their type of contract; admission policies and policies for student guidance; planning of study programmes; teaching methodologies and student assessment methods; opportunities for mobility; students’ rights and duties, including the amount of fees and other payments necessary; student social support services; mechanisms to deal with complaints and suggestions; access to material resources and teaching support; teaching outcomes, expressed in the academic results, integration in the labour market and level of satisfaction of stakeholders; internal quality assurance policies, accreditation certificates, and results of evaluation of the institution and of its study cycles.

103

8.2 CORE ELEMENTS FOR THE CERTIFICATION OF INTERNAL QUALITY ASSURANCE SYSTEMS

The comparative analysis undertaken in 7.2 provides some important information concerning the characteristics of the main models of institutional audit operating in Europe, the components which they share and others which distinguish them, and also the concern that these models should be seen not as mere mechanisms of external control, although this aim is present in some models of compulsory audit, but also, or mainly, as encouragement for an internal culture within the higher education institutions which opts for quality as a strategic objective which is central to the mission of the institution.

The opinions expressed are generally positive regarding the acceptance of institutional audit processes on the part of institutions and their dynamic impact on the development of internal quality assurance systems. In two of the countries, where one complete cycle of audits has already been completed, there is mention of the evident success of this exercise in areas related to student involvement and to a better involvement of staff in enhancement of teaching and learning and in the strategic management of quality enhancement in institutions195 (Scotland). There is also the perception that the criteria established for the audits, developed in cooperation with higher education institutions, contributed to the development of quality in education196 (Norway).

In the design of an institutional audit process, it is worth remembering the weariness of the institutions in relation to the effort which has come to be demanded from them in the area of quality assurance, not always with obvious effects or advantages. As is stated in Trends V, “external quality assurance systems also need to demonstrate that they actually produce an improvement in quality. Considerable concern still remains about the increasing bureaucratic burden on institutions”197. The ready acceptance of the processes of institutional audit which has been seen in many countries is, in part, related to the expectation that they may lead to lighter external quality assurance procedures, to counterbalance the greater attention to internal quality assurance processes, which hopefully should be oriented towards creativity and innovation in the achievement of the mission of the institution198, promoting improvement of processes and results ,

195 Brink, C. & Kohler, A. (2009). Audit Approaches – Features for a Common Understanding. In AQA (2009b), p. 39. 196 Ibidem, p. 40. 197 Crosier, D. et al. (2007). Trends V: Universities Shaping the European Higher Education Area, p. 9. 198 See conclusion of the Project QAHECA (point 5.4).

104

without affecting a transparent provision of information to stakeholders and society in general199.

The pedagogic and formative role of quality assurance is another aspect to be valorized, particularly in relation to institutions which are still in the early stages of development of internal quality systems. Institutional audit processes more geared towards enhancement of quality, and which include a component of guidance and support to the institution, tend to reinforce this pedagogic role, encouraging and making it easier for institutions to take responsibility for quality assurance.

The development of a model for institutional audit implies the previous definition of standards, which at the same time constitute a reference for the institutions in the design and development of their internal quality assurance systems, and for the auditors in their scrutiny of these systems. The definition of standards should be undertaken in a sufficiently open way, so that so that they do not encourage mere conformity, and should be participated by the higher education institutions, so that they may become more attuned to the real significance and scope of the standards and may develop a sense of ownership, which increases acceptance of the model and motivation for their effective involvement in the work to be done. This concern should also be present even when the aim of the audit is a formal decision about certification of the internal quality assurance system of the institution, insofar as, without due care, “the certification model with its emphasis on procedure fulfilment, however, may have the side-effect of increasing bureaucratisation and causing process compliance, with a view of professionalism based on lists of skills and achievements to be determined externally and this might exclude reflection”200.

In conclusion, in the face of the previous reflections, the comparative analysis undertaken in 7.2 and also those included in 8.1 on the internal quality systems, without forgetting other relevant points raised throughout this study, namely the conclusions of the QAHECA Project, it is possible to summarise some key aspects to be taken into consideration in the planning and implementation of a model for certification of internal quality assurance systems in higher education through process of institutional audit, in line with what is beginning to be defined as European good practices in this field:

199 A challenge facing quality assurance systems is indeed finding a balance between the interest of the institutions in lighter external evaluation mechanisms and the interest of society, in particular of students, in having transparent information on the quality of study programmes. As was said with regard to the German model, “the intriguing question (...) is how a productive interplay between internal and external governance of quality can be achieved and to what extent European HE systems can come together in this regard” (Witte, 2008, p. 52). 200 Smith, Armstrong and Brown (1999), cited by Fernández et al. (2008), p. 61.

105

- Cooperation and dialogue – Quality assurance is context-sensitive. The preparation of the design of the institutional audit model should therefore include extensive consultation with stakeholders, with a view to ensuring the adaptation of the model to the needs felt by the institution and other relevant actors, as well as the acceptance and interested participation of the institutions. This concern is particularly pertinent in the definition of the standards for certification, because of its decisive role in the actual development of internal quality assurance systems.

- Social acceptance – Social pressures have come to influence quality assurance systems, not always in a rational way or even for the promotion of quality. It is important that the consultation process also attempts to ensure the acceptance, on the part of society and the political powers, of the idea that institutional audit is a good approach to external quality assurance, at least as credible and as effective as accreditation of study programmes.

- Clarity of aim – The aims of the model of institutional audit to be developed should be clearly defined, namely with regard to the balance to be established between the function of control and confirmation of compliance, and the purpose for continuous enhancement of quality.

- Lightening of the processes – A trend can be seen in Europe for systems of institutional audit either to be the only process of external quality assurance or, when they are cumulative with a programme accreditation process, to contribute to their marked simplification. The interaction between institutional audit processes and study programme accreditation should, therefore, be defined and made very clear from the outset (as part of the clarity of aims), and it is desirable that the certification of the internal quality assurance system of an institution by the Agency should lead to a significant reduction in effort associated with individual accreditation of its study programmes.

- Promotion of enhancement – Experience gained from case studies emphasises the positive effects of the promotion and enhancement of quality as a result of audits which focus on the encouragement of creativity, innovation and risk assumption, to the detriment of strict compliance with externally defined standards. The model of institutional audit to be installed should take this into account, both in the definition of aims and in its form of organisation, in order to inspire a true culture of quality in the institutions.

- Pedagogic and formative role – In line with this idea, it is desirable that the audit process should have a pedagogic and formative component, as a contribution to improving internal organisation of quality assurance procedures in institutions, expressed, namely, in the type of information and guidelines made available by the agencies, in support from and contacts with specialists from the Agency and in recommendations made in the audit reports.

- Non-prescriptive standards – The institutional audit process should not encourage standardised models of internal quality assurance, which would naturally tend to favour formal compliance. The audit standards should therefore be formulated in a sufficiently open way, they should not be prescriptive with regard to the concrete way the internal quality systems are

106

implemented, and should be accompanied by guidelines to support the institutions in the development of these systems.

- References for the standards – The ESG and the national juridical framework constitute an obligatory reference for the establishment of standards. The set of references suggested in 8.1.1 exemplifies a possible approach to the definition of standards.

- Simplification of procedures – If the proper care is not taken, quality assurance processes tend to generate bureaucracy. The procedures associated to the institutional audit process should preferably be designed in the simplest possible way, with the aim of supporting and assisting the institution in the development of its initiatives, without imposing an onerous bureaucratic framework which is foreign to their institutional culture201.

- Preparation of the audit – The initial contacts between the Agency and the institution for the preparation of the audit are an excellent forum for defining the line of dialogue and collaboration between the agencies and the institutions. Previous procedures for the preparation of the audit, as well as guiding and supporting the institution in the work of preparation for the audit, can also help in raising awareness among the academic community of the advantages of internal quality processes and the need for their effective development. Examples of good practices referred to previously include initial dialogue with the institution, provision of sufficient information on the part of the Agency, the organisation of previous widely-attended meetings or seminars and a formal contract setting out the terms of reference for the audit.

- Participation of stakeholders – The central role of students and the effective participation of internal and external stakeholders in internal quality processes are widely considered to be essential elements in an institutional quality culture. This should be remembered when standards for the audit process are defined, following the example of what happens in the various audit systems set up in Europe.

- Follow-up – The establishment of follow-up mechanisms is important for the credibility of the audit processes and also as a means of continuing support to the development of institutions. There are advantages to be gained when the follow-up procedures are included from the start as an integral part of the global design of the audit process, particularly if they are based on some of the good practices referred to previously in relation to systematic and constructive dialogue with the institutions.

- Dissemination of global results and good practices – Another aspect of systematic follow-up is the processing of the audit reports (annually or by assessment cycles) to identify important common aspects and good practices, which should be disseminated as a contribution to the overall improvement of

201 For example, the self-evaluation reports for the purpose of audit may be considerably simplified if the institution is encouraged to present pre-existing documentation on the quality system, its operation and results. An example of good practice here is the system in operation in Norway, in which the NOKUT Agency does not ask for a self-evaluation report, because the whole of the audit is based on documentation which is part of the actual internal quality system.

107

the higher education system and the system of quality assurance. The design of the audit system to be developed should include this type of mechanism of processing and disseminating overall results and good practices.

8.3 SUGGESTIONS FOR FUTURE WORK

The decision of the A3ES to install a process of institutional audit for certification of internal quality assurance systems in higher education institutions, as part of the intention to establish lighter procedures for study programme accreditation, as an alternative to the current model of individual accreditation of study programmes, clearly follows the most up to date European trends for external quality assurance, as stands out from this study, which analyses the evolution of quality assurance in the European Higher Education Area, particularly regarding the development of institutional audit models.

Before certification, a fundamental question is raised concerning the specification of internal systems of quality assurance, in face of the European and national standards. This is an indispensable starting point, so that institutions may prepare for participation in the audit process, defining and implementing their strategies and procedures for internal quality assurance, in compliance with the general criteria established by the Agency. As has been reiterated in this work, good practices point out to the advantages of shared work for construction of the reference framework to be adopted, involving the Agency, the institutions and other relevant stakeholders, namely the political powers and professional associations. The suggestion of reference points presented in 8.1.1, which has the agreement, in principle, of the Agency, constitutes a starting point for consultation and dialogue with the stakeholders.

At the same time, the question arises concerning the general architecture and underpinning principles of the certification model to be created. The suggestions put forward in 8.2 could be a catalyst in the process of reflection with stakeholders, in such a way that the Agency may establish the foundation of the audit process with greater likelihood of both social acceptance of the model and the constructive and interested participation of the institutions.

Once the standards and general principles to be observed have been established, it is necessary to work on the procedures and documentation for institutional audit, with emphasis on preparation of the respective handbook and necessary guidelines. In the ambit of such a task, there are concrete aspects of a methodological nature to be considered, which have not been dealt with in this study, as they are essentially of an operational nature, such as, for example, the profile of the auditors and the composition

108

of the audit teams, the training of auditors, the type and organization of site visits, and the type, contents and dissemination of final reports. With some extra work in the area of data collection and processing, it will be possible to gather a good deal of information and knowledge from the case studies referred to.

Thus, a framework was established for the tasks to be performed in the near future, in order to follow the positive direction indicated by the Agency A3ES, in such a way as to lead the national quality assurance system towards levels of modernity which will place it among the European systems of reference. There are, however, a few final observations which are worth making, due to the impact they may have on the work of reflection and implementation which approaches.

The first of these is that quality assurance is a costly process, and it is illusory to believe that the transition to an institutional audit system will imply a global reduction of costs, insofar as the installation and operation of internal quality systems will require considerable resources202. As was said in 6.1, in the majority of European countries considerable financial incentives were introduced for the improvement of internal quality assurance systems in the institutions, as recognition of the financial effort involved in this work. The acknowledgement of this question and the subsequent provision of measures of support and encouragement are essential so that Portugal may be in the first ranks of quality and quality assurance in European higher education.

A second idea has to do with the need for long-term commitment to quality, as a key factor of success for a quality culture – “the development and nourishment of quality cultures does not happen overnight: the dynamics of real enhancement depends to a significant extent on operating within a relatively stable, predictable policy environment”203. The most recent experiences of installing institutional audit processes with the inclusion of an institutional support and/or consultancy phase204 show that the development of internal quality systems takes time and requires an experimental and consolidation period before it is prepared to formally undergo a certification process.

And finally, but no less important, is the question of keeping the quality assurance system sufficiently light, particularly regarding the frequency of assessments, so that,

202 In the words of Johanna Witte, a researcher at the Bavarian State Institute for Higher Education Research and Planning, when referring to the German model for system accreditation, “the resources cannot be saved; they need to be shifted from the end of the pipeline to the beginning of the pipeline”, in an allusion to the financial burdens which are transferred from the Agencies to the higher education institutions. 203 AQA (2009b). Trends of Quality Assurance and Quality Management in Higher Education Systems, p. 61, in a reference to the experience gained from the programme Quality Enhancement Framework, in Scotland. 204 The cases of Spain and Austria.

109

without detriment to the achievement of the core objectives of the system, routine and weariness with quality assurance are avoided in the institutions. Indeed, as Sybille Reichart wonders: “With the increasing frequency of quality reviews, how can one prevent routine from setting in and undermining the motivation to invest the quality assurance with a genuine desire to identify one’s weaknesses and to improve?”205. At least part of the answer is provided by Ko Scheele: “The quest for the Holy Grail of optimum quality assurance is more about smart systems than about large ones”206.

205 Reichert, S. (2008). Looking Back – Looking Forward: Quality Assurance and the Bologna Process. In EUA (2008), p. 7. 206 Scheele, K. (2004). Licence to Kill: About Accreditation Issues and James Bond. In Di Nauta et al. (2004), p. 22. Ko Scheele is an inspector at the Inspectorate of Education in the Netherlands.

110

BIBLIOGRAPHY

A3ES (2009). Activity Plan for 2009, Agency for the Assessment and Accreditation of Higher Education, Lisbon.

Aas, G., Askling, B., Dittrich, K., Froestad, W., Haugh, P., Lycke, K., Moitus, S., Pyykö, R. & Sörskår, A. (2009). Assessing Education Quality: Knowledge Production and the Role of Experts, ENQA Workshop Reports 6, European Association for Quality Assurance in Higher Education, Helsinki.

Amaral, A. & Rosa, M.J. (2008). International Review of the Portuguese Quality Assurance System. In EUA (Ed.), Implementing and Using Quality Assurance: Strategy and Practice – A Selection of Papers from the 2nd European Quality Assurance Forum, EUA Case Studies 2008, European University Association, Brussels, pp. 74-79.

ACQUIN (2004). Pilot Project “Process Quality for Teaching and Learning” – Concept and Implementation of a Process Accreditation Approach, Accreditation, Certification and Quality Assurance Institute, Bayreuth.

ACQUIN (2009). Guidelines for System Accreditation Procedures, Accreditation, Certification and Quality Assurance Institute, Bayreuth. http://www.acquin.org/doku_serv/ Guide_System_EN_ACQUIN.pdf, retrieved 27.10.2009.

ANECA (2006). Propuesta – Criterios y Directrices para la Acreditación de Enseñanzas Universitarias Conducentes a Títulos Oficiales Españoles de Grado e Máster, Ed:01 12/05/06, Agencia Nacional de Evaluación de la Calidad y Acreditación, Madrid. http://www.aneca.es/media/168582/pa_modelo_061213.pdf, retrieved 09.11.2009.

ANECA (2009a). Memoria de Actividades Enero - Diciembre 2008, Agencia Nacional de Evaluación de la Calidad y Acreditación, Abril 2009, Madrid. http://www.aneca.es/media/316317/publi_memoria08.pdf, retrieved 09.11.2009.

ANECA (2009b). Plan de Actuación 09, Agencia Nacional de Evaluación de la Calidad y Acreditación, Madrid. http://www.aneca.es/media/142706/plandeactuacion_2009_090128.pdf, retrieved 09.11.2009.

ANECA, AQU & ACSUG (2007a). PROGRAMA AUDIT: Guía para el diseño de Sistemas de Garantía Interna de Calidad de la formación universitaria, Documento 01, v. 1.0_21/06/07, Madrid. http://www.aneca.es/media/166338/audit_doc01_guidiseno_070621.pdf, retrieved 21.10.2007.

ANECA, AQU & ACSUG (2007b). PROGRAMA AUDIT: Directrices, definición y documentación de Sistemas de Garantía Interna de Calidad de la formación universitaria, Documento 02, v. 1.0_21/06/07, Madrid. http://www.aneca.es/media/166342/ audit_doc02_directrices_070621.pdf, retrieved 21.10.2007.

ANECA, AQU & ACSUG (2007c). PROGRAMA AUDIT: Herramientas para el Diagnóstico en la implantación de Sistemas de Garantía Interna de Calidad de la formación universitaria, Documento 03, v. 1.0_06/07/07, Madrid. http://www.aneca.es/ media/166346/audit_doc03_herramientas_070621.pdf, retrieved 21.10.2007.

ANECA, AQU & ACSUG (2008). PROGRAMA AUDIT: Guía de Evaluación del diseño de Sistemas de Garantía Interna de Calidad de la formación universitaria,

111

Documento 04, v. 02_13/02/08, Madrid. http://www.aneca.es/media/166350/audit_doc04_ guiaevaluacion_080219.pdf, retrieved 09.11.2009.

AQA (2009a). AQA Quality Audits – Accreditation of the Quality Management of Performance Areas at the Higher Education Institutions, Austrian Agency for Quality Assurance, Vienna. Retrieved from http://www.aqa.ac.at, 27.10.2009.

AQA (2009b). Trends of Quality Assurance and Quality Management in Higher Education Systems, Austrian Agency for Quality Assurance, Vienna. http://www.aqa.ac.at/ download.483.trends-of-qa-and-qm-in-hei-2009.pdf, retrieved 27.10.2009.

Bozo, D., Damian, R., González, C., Helle, E., Imoda, F., Kohler, A., Papazoglou, V., Dalmau, G. & Shopov, T. (2009). Current Trends in European Quality Assurance, ENQA Workshop Reports 8, European Association for Quality Assurance in Higher Education, Helsinki.

Brink, C. & Kohler, A. (2009). Audit Approaches – Features for a Common Understanding. In AQA (Ed.), Trends of Quality Assurance and Quality Management in Higher Education Systems, Austrian Agency for Quality Assurance, Vienna, pp. 35-55.

Carney, C. & Macpherson, C. (2009). Scotland’s Quality Enhancement Themes: a creative approach engaging the sector and embracing institutional diversity, presentation at the 4th European Quality Assurance Forum, Copenhagen, November 2009. Retrieved from www.eua.be/events/eqaf-copenhagen/presentations.

Codina, E. A. & Jiménez, E. G. (2008). Evaluation of teacher competence in Spain: The DOCENTIA programme. In EUA (Ed.), Implementing and Using Quality Assurance: Strategy and Practice – A Selection of Papers from the 2nd European Quality Assurance Forum, EUA Case Studies 2008, European University Association, Brussels, pp. 17-22.

Costes, N., Crozier, F., Cullen, P., Grifoll, J., Harris, N., Helle, E., Hopbach, A., Kekäläinen, H., Knezevic, B., Sits, T. & Sohm, K. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, ENQA Occasional Papers 14, Helsinki.

Council of the European Union (1998). Recommendation 98/561/CE, 24th September, 1998, regarding European cooperation, with a view to assuring quality in higher education, Official Journal of the European Communities 270, 07.10.1998.

Criteria for Evaluation of Universities and University Colleges’ Quality Systems for Educational Activities. NOKUT, Oslo. http://www.nokut.no/sw481.asp, retrieved 27.10.2009.

Crosier, D., Purser, L. & Smidt, H. (2007). Trends V: Universities Shaping the European Higher Education Area – an EUA Report, European University Association, Brussels.

Crozier, F., Curvale, B., Dearlove, R., Helle, E. & Hénard, F. (2006). Terminology of quality assurance: towards shared European values?, ENQA Occasional Papers 12, Helsinki.

Crozier, F., Curvale, B. & Hénard, F. (2006). Final report on the pilot Quality Convergence II project: Promoting epistemological approaches to quality

112

assurance. In F. Crozier, B. Curvale, R. Dearlove, E. Helle & F. Hénard (Eds.), Terminology of quality assurance: towards shared European values?, ENQA Occasional Papers 12, Helsinki, pp. 22-26.

CRUP (2003). Guiding Principles for the Evaluation and Follow-up of Activities within Higher Education Institutions (translated title), document adopted at the meeting of the Protuguese Rectors’ Conference on the 21st September, 1993, registered in Minutes nr. 140, Annex II, Lisbon.

Dalmau, G. R. & González, C. R. (2009). ESG and the Current QA Trends in Spain: more than just a change. In D. Bozo, R. Damian, C. González, E. Helle, F. Imoda, A. Kohler, V. Papazoglou, G. Dalmau & T. Shopov (Eds.), Current Trends in European Quality Assurance, ENQA Workshop Reports 8, European Association for Quality Assurance in Higher Education, Helsinki, pp. 14-18.

Davidson, A. (2009). Future Directions for the Scottish Enhancement-led Approach to Quality. In EUA (Ed ), Trends in Quality Assurance – A Selection of Papers from the 3rd European Quality Assurance Forum, EUA Case Studies 2009, European University Association, Brussels, pp. 30-38.

Di Nauta, P., Omar, P.-L., Schade, A. & Scheele, J.P. (Eds.), Accreditation Models in Higher Education – Experiences and Perspectives, ENQA Workshop Reports 3, Helsinki.

Dittrich, K. (2004). Accreditation in the Netherlands. In P. Di Nauta, P.-L. Omar, A. Schade & J. Scheele (Eds.), Accreditation Models in Higher Education – Experiences and Perspectives, ENQA Workshop Reports 3, Helsinki, pp. 55-58.

EC(2009). Higher Education in Europe 2009: Developments in the Bologna Process, Education, Audiovisual and Culture Executive Agency (P9 Eurydice), European Commission, Brussels, March 2009.

ENQA (2003). Quality Procedures in European Higher Education – An ENQA Survey, ENQA Occasional Papers 5, European Network for Quality Assurance in Higher Education, Helsinki.

ENQA (2005). Standards and Guidelines for Quality Assurance in the European Higher Education Area, European Association for Quality Assurance in Higher Education, Helsinki.

ENQA (2006). Guidelines for national reviews of ENQA member agencies, ENQA Board, 21 September 2006, European Association for Quality Assurance in Higher Education, Helsinki.

ENQA (2009). ENQA Position Paper on Quality Assurance in the EHEA in view of the Leuven and Louvain-la-Neuve meeting of ministers responsible for higher education of 28-29 April 2009, 4 March 2009, European Association for Quality Assurance in Higher Education, Helsinki.

Enroth, I. (2009). Audits in Denmark. In AQA (Ed.), Trends of Quality Assurance and Quality Management in Higher Education Systems, Austrian Agency for Quality Assurance, Vienna, pp. 95-99.

Erichsen, H.-U. (2004). Quality Assurance and Accreditation in Germany. In P. Di Nauta, P.-L. Omar, A. Schade & J. Scheele (Eds.), Accreditation Models in

113

Higher Education – Experiences and Perspectives, ENQA Workshop Reports 3, Helsinki, pp. 42-46.

EUA (2003). Graz Declaration 2003 – Forward from Berlin: the Role of the Universities, European University Association, Brussels, September 2003.

EUA (2005). Developing an Internal Quality Culture in European Universities, Report on the Quality Culture Project 2002-2003, EUA Publications 2005, European University Association, Brussels.

EUA (2006). Quality Culture in European Universities: A Bottom-Up Approach, Report on the three rounds of the Quality Culture Project 2002-2006, EUA Publications 2006, European University Association, Brussels.

EUA (2007a). Embedding Quality Culture in Higher Education – A Selection of Papers from the 1st European Quality Assurance Forum, EUA Case Studies 2007, European University Association, Brussels.

EUA (2007b). Creativity in Higher Education, Report on the EUA Creativity Project 2006-2007, EUA Publications 2007, European University Association, Brussels.

EUA (2008). Implementing and Using Quality Assurance: Strategy and Practice – A Selection of Papers from the 2nd European Quality Assurance Forum, EUA Case Studies 2008, European University Association, Brussels.

EUA (2009a). Improving Quality, Enhancing Creativity: Change Processes in European Higher Education Institutions, Final Report of the Quality Assurance for the Higher Education Change Agenda (QAHECA) Project, EUA Publications 2009, European University Association, Brussels.

EUA (2009b). Trends in Quality Assurance – A Selection of Papers from the 3rd European Quality Assurance Forum, EUA Case Studies 2009, European University Association, Brussels.

Externally Reviewed Agencies. ENQA, Helsinki. http://www.enqa.eu/reviews_agencies.lasso, retrieved 14.05.2010.

Fernández, S., Esteban Fernández, S. & Álvarez, A. (2008). University Institutional Evaluation and Academic Achievement. In EUA (Ed.), Implementing and Using Quality Assurance: Strategy and Practice – A Selection of Papers from the 2nd European Quality Assurance Forum, EUA Case Studies 2008, European University Association, Brussels, pp. 60-67.

FINHEEC (2007). Audits of Quality Assurance Systems of Finish Higher Education Institutions. Audit Manual for 2008-2011. Publications of Finnish Higher Education Evaluation Council 10:2007.

Froestad, W. (2009). Norway. In AQA (Ed.), Trends of Quality Assurance and Quality Management in Higher Education Systems, Austrian Agency for Quality Assurance, Vienna, pp. 76-82.

GAC (2008). Criteria for System Accreditation, Printed Matter 11/2008 (resolved on 08 October 2007, amended on 29 February 2008), German Accreditation Council, Bonn.

114

Green, R. (2005). National Reports 2004-2005. England, North Ireland and Wales. http://www.ond.vlaanderen.be/hogeronderwijs/bologna/links/National-reports2005/National_Report_UK-England-Wales-N-Ireland%202005.pdf, retrieved 27.10.2009.

Hanft, A. & Kohler, A. (2008). How Can External Quality Assurance Support Institutional Quality Management. In EUA (Ed.), Implementing and Using Quality Assurance: Strategy and Practice – A Selection of Papers from the 2nd European Quality Assurance Forum, EUA Case Studies 2008, European University Association, Brussels, pp. 53-59.

Harvey, L. (2008). Using the European Standards and Guidelines: Some Concluding Remarks. In EUA (Ed.), Implementing and Using Quality Assurance: Strategy and Practice – A Selection of Papers from the 2nd European Quality Assurance Forum, EUA Case Studies 2008, European University Association, Brussels, pp. 80-85.

Hazelkorn, E. (2009). The Emperor has no Clothes? Rankings and the shift from quality assurance to world-class excellence. In EUA (Ed.), Trends in Quality Assurance – A Selection of Papers from the 3rd European Quality Assurance Forum, EUA Case Studies 2009, European University Association, Brussels, pp. 10-18.

Hodgson, K. (2009). Developments to Encourage Quality Enhancement: A Case Study. In EUA (Ed.), Trends in Quality Assurance – A Selection of Papers from the 3rd European Quality Assurance Forum, EUA Case Studies 2009, European University Association, Brussels, pp. 19-23.

Holmen, T.B. (2004). Quality Assurance of Norwegian Higher Education. In P. Di Nauta, P.-L. Omar, A. Schade & J. Scheele (Eds.), Accreditation Models in Higher Education – Experiences and Perspectives, ENQA Workshop Reports 3, Helsinki, pp. 59-61.

Hopbach, A. (2009). Germany. In AQA (Ed.), Trends of Quality Assurance and Quality Management in Higher Education Systems, Austrian Agency for Quality Assurance, Vienna, pp. 83-87.

HRK (1995). Resolution: Evaluation in German Higher Education Institutions with particular reference to the evaluation of teaching, German Rectors’ Conference, Bonn.

HRK (2007). The Quality Assurance System for Higher Education at European and National Level, Bologna Seminar Berlin, 15/16 February 2007, German Rectors’ Conference, Bonn.

IWA 2 (2007). Quality Management Systems – Guidelines for the application of ISO 9001:2000 in education, International Workshop Agreement, ISO, Geneva.

Klaassen, L. (2009). NVAO: Introducing, developing and implementing a new phase in the accreditation system in the Netherlands and Flanders. In EUA (Ed.), Improving Quality, Enhancing Creativity: Change Processes in European Higher Education Institutions, Final Report of the Quality Assurance for the Higher Education Change Agenda (QAHECA) Project, EUA Publications 2009, European University Association, Brussels, pp. 38-41.

Kohler, A. (2009). Quality Assurance in Austrian Higher Education – Features and Challenges. In D. Bozo, R. Damian, C. González, E. Helle, F. Imoda, A. Kohler,

115

V. Papazoglou, G. Dalmau & T. Shopov (Eds.), Current Trends in European Quality Assurance, ENQA Workshop Reports 8, European Association for Quality Assurance in Higher Education, Helsinki, pp. 9-13.

Maurer, S. & Beccari, L. (2009). Quality Audits in Switzerland. In AQA (Ed.), Trends of Quality Assurance and Quality Management in Higher Education Systems, Austrian Agency for Quality Assurance, Vienna, pp. 62-65.

Moitus, S. & Seppälä, H. (2009). Audits of Quality Assurance Systems of Finish Higher Education Institutions. In AQA (Ed.), Trends of Quality Assurance and Quality Management in Higher Education Systems, Austrian Agency for Quality Assurance, Vienna, pp. 66-71.

OAQ (2005). Summary Quality Assurance Audits According to the UFG – OAQ Synthesis Report, Centre of Accreditation and Quality Assurance of the Swiss Universities, January 2005, Bern. http://www.oaq.ch/pub/en/documents/Leitfaden_E_2007-09-20.pdf, retrieved 09.12.2009.

OAQ (2007). Quality Audits 2007/08 – Concept, Procedure and Quality Standards: Guide for Universities, Centre of Accreditation and Quality Assurance of the Swiss Universities, January 2005, Bern. http://www.oaq.ch/pub/downloads/Anerkennung_e.pdf, retrieved 09.12.2009.

QAA (2006). Handbook for Institutional Audit: England and Northern Ireland 2006, The Quality Assurance Agency for Higher Education, Gloucester. http://www.qaa.ac.uk/reviews/institutionalAudit/handbook2006/Handbook2006.pdf, retrieved 30.10.2009.

QAA (2008). Enhancement-led Institutional Review Handbook: Scotland (2nd edition), The Quality Assurance Agency for Higher Education, Gloucester. http://www.qaa.ac.uk/reviews/ELIR/handbook08final/ELIRHandbook2008.pdf, retrieved 30.10.2009.

QAA (2009). Handbook for Institutional Audit: England and Northern Ireland 2009, The Quality Assurance Agency for Higher Education, Gloucester. http://www.qaa.ac.uk/reviews/institutionalAudit/handbook2009/InstitutionalAuditHandbook2009.pdf, retrieved 30.10. 2009.

Rauhvargers, A., Deane, C. & Pawels, W. (2009). Bologna Process Stocktaking Report 2009, Report from working groups appointed by the Bologna Follow-up Group to the Ministerial Conference in Leuven/Louvain-la-Neuve 28-29 April 2009, BFUG.

Realising the European Higher Education Area (2003). Communiqué of the Conference of Ministers responsible for Higher Education, Berlin, 19 September 2003.

Reichert, S. & Tauch, C. (2005). Trends IV: European Universities Implementing Bologna, European University Association, Brussels.

Reichert, S. (2008). Looking Back – Looking Forward: Quality Assurance and the Bologna Process. In EUA (Ed.), Implementing and Using Quality Assurance: Strategy and Practice – A Selection of Papers from the 2nd European Quality Assurance Forum, EUA Case Studies 2008, European University Association, Brussels, pp. 5-10.

116

Santos, S.M. (Rapporteur), Gonçalves, L.C., Silva, J.D., Fonseca, L.A., Filipe, A.F., Vieira, C., Lima, M.J. & Oliveira, M.F. (2006). Review of the Quality Assurance and Accreditation Policies and Practices in the Portuguese Higher Education – Self-Evaluation Report, CNAVES, FUP, ADISPOR, APESP, Lisbon.

Scheele, K. (2004). Licence to Kill: About Accreditation Issues and James Bond. In P. Di Nauta, P.-L. Omar, A. Schade & J.P. Scheele (Eds.), Accreditation Models in Higher Education – Experiences and Perspectives, ENQA Workshop Reports 3, Helsinki, pp. 19-25.

Serrano-Velarde, K. & Hopbach, A. (2007). European politics of quality assurance and the introduction of the Standards and Guidelines for Quality Assurance in the European Higher Education Area. In HRK (Ed.), The Quality Assurance System for Higher Education at European and National Level, Bologna Seminar Berlin, 15/16 February 2007, German Rectors’ Conference, Bonn, pp. 29-62.

Simão, J.V., Santos, S.M. & Costa, A.A. (2005). Ambition to Excellence – The Bologna Opportunity (translated title), Gradiva, Lisbon.

The Bologna Process 2020 – The European Higher Education Area in the new decade (2009). Communiqué of the Conference of European Ministers Responsible for Higher Education, Leuven and Louvain-la-Neuve, 28-29 April 2009.

The European Higher Education Area (1999). Joint Declaration of the European Ministers of Education Convened in Bologna on the 19th of June 1999.

The European Higher Education Area – Achieving the Goals (2005). Communiqué of the Conference of European Ministers Responsible for Higher Education, Bergen, 19-20 May 2005.

Towards the European Higher Education Area (2001). Communiqué of the meeting of European Ministers in charge of Higher Education in Prague on May 19th 2001.

Towards the European Higher Education Area: responding to challenges in a globalised world (2007). London Communiqué, 18 May 2007.

van Galen, S. & Woutersen, M. (2009). Balancing Quality Enhancement and Accountability – Reforming the Dutch and Flemish Accreditation System, paper presented at the 4th European Quality Assurance Forum, Copenhagen, November 2009. Retrieved from www.eua.be/events/eqaf-copenhagen/presentations.

Witte, J. (2008). The Changing Political Framework of Quality Assurance in German Higher Education: National Debates in the European Context. In EUA (Ed.), Implementing and Using Quality Assurance: Strategy and Practice – A Selection of Papers from the 2nd European Quality Assurance Forum, EUA Case Studies 2008, European University Association, Brussels, pp. 48-52.

117

ANNEX – PART 1 OF THE EUROPEAN STANDARDS AND GUIDELINES

Part 1: European standards and guidelines for internal quality assurance within higher

education institutions 1.1 Policy and procedures for quality assurance

STANDARD: Institutions should have a policy and associated procedures for the assurance of the quality

and standards of their programmes and awards. They should also commit themselves explicitly to the development of a culture which recognises the importance of quality, and quality assurance, in their work. To achieve this, institutions should develop and implement a strategy for the continuous enhancement of quality.

The strategy, policy and procedures should have a formal status and be publicly available. They should also include a role for students and other stakeholders.

GUIDELINES: Formal policies and procedures provide a framework within which higher education

institutions can develop and monitor the effectiveness of their quality assurance systems. They also help to provide public confidence in institutional autonomy. Policies contain the statements of intentions and the principal means by which these will be achieved. Procedural guidance can give more detailed information about the ways in which the policy is implemented and provides a useful reference point for those who need to know about the practical aspects of carrying out the procedures.

The policy statement is expected to include:

• the relationship between teaching and research in the institution; • the institution’s strategy for quality and standards; • the organisation of the quality assurance system; • the responsibilities of departments, schools, faculties and other organisational units and

individuals for the assurance of quality; • the involvement of students in quality assurance; • the ways in which the policy is implemented, monitored and revised.

The realisation of the EHEA depends crucially on a commitment at all levels of an institution to ensuring that its programmes have clear and explicit intended outcomes; that its staff are ready, willing and able to provide teaching and learner support that will help its students achieve those outcomes; and that there is full, timely and tangible recognition of the contribution to its work by those of its staff who demonstrate particular excellence, expertise and dedication. All higher education institutions should aspire to improve and enhance the education they offer their students. 1.2 Approval, monitoring and periodic review of programmes and awards

STANDARD: Institutions should have formal mechanisms for the approval, periodic review and

monitoring of their programmes and awards.

GUIDELINES: The confidence of students and other stakeholders in higher education is more likely to be

established and maintained through effective quality assurance activities which ensure that programmes are well-designed, regularly monitored and periodically reviewed, thereby securing their continuing relevance and currency.

The quality assurance of programmes and awards are expected to include:

• development and publication of explicit intended learning outcomes;

118

• careful attention to curriculum and programme design and content; • specific needs of different modes of delivery (e.g. full time, part-time, distance-learning,

e-learning) and types of higher education (e.g. academic, vocational, professional); • availability of appropriate learning resources; • formal programme approval procedures by a body other than that teaching the

programme; • monitoring of the progress and achievements of students; • regular periodic reviews of programmes (including external panel members); • regular feedback from employers, labour market representatives and other relevant

organisations; • participation of students in quality assurance activities.

1.3 Assessment of students

STANDARD: Students should be assessed using published criteria, regulations and procedures which are

applied consistently.

GUIDELINES: The assessment of students is one of the most important elements of higher education. The outcomes of assessment have a profound effect on students’ future careers. It is

therefore important that assessment is carried out professionally at all times and that it takes into account the extensive knowledge which exists about testing and examination processes. Assessment also provides valuable information for institutions about the effectiveness of teaching and learners’ support.

Student assessment procedures are expected to:

• be designed to measure the achievement of the intended learning outcomes and other programme objectives;

• be appropriate for their purpose, whether diagnostic, formative or summative; • have clear and published criteria for marking; • be undertaken by people who understand the role of assessment in the progression of

students towards the achievement of the knowledge and skills associated with their intended qualification;

• where possible, not rely on the judgements of single examiners; • take account of all the possible consequences of examination regulations; • have clear regulations covering student absence, illness and other mitigating

circumstances; • ensure that assessments are conducted securely in accordance with the institution’s

stated procedures; • be subject to administrative verification checks to ensure the accuracy of the

procedures.

In addition, students should be clearly informed about the assessment strategy being used for their programme, what examinations or other assessment methods they will be subject to, what will be expected of them, and the criteria that will be applied to the assessment of their performance.

1.4 Quality assurance of teaching staff

STANDARD: Institutions should have ways of satisfying themselves that staff involved with the teaching

of students are qualified and competent to do so. They should be available to those undertaking external reviews, and commented upon in reports.

119

GUIDELINES: Teachers are the single most important learning resource available to most students. It is important that those who teach have a full knowledge and understanding of the subject

they are teaching, have the necessary skills and experience to transmit their knowledge and understanding effectively to students in a range of teaching contexts, and can access feedback on their own performance. Institutions should ensure that their staff recruitment and appointment procedures include a means of making certain that all new staff have at least the minimum necessary level of competence. Teaching staff should be given opportunities to develop and extend their teaching capacity and should be encouraged to value their skills. Institutions should provide poor teachers with opportunities to improve their skills to an acceptable level and should have the means to remove them from their teaching duties if they continue to be demonstrably ineffective.

1.5 Learning resources and student support

STANDARD: Institutions should ensure that the resources available for the support of student learning are

adequate and appropriate for each programme offered.

GUIDELINES: In addition to their teachers, students rely on a range of resources to assist their learning.

These vary from physical resources such as libraries or computing facilities to human support in the form of tutors, counsellors, and other advisers. Learning resources and other support mechanisms should be readily accessible to students, designed with their needs in mind and responsive to feedback from those who use the services provided. Institutions should routinely monitor, review and improve the effectiveness of the support services available to their students.

1.6 Information systems

STANDARD: Institutions should ensure that they collect, analyse and use relevant information for the

effective management of their programmes of study and other activities.

GUIDELINES: Institutional self-knowledge is the starting point for effective quality assurance. It is

important that institutions have the means of collecting and analysing information about their own activities. Without this they will not know what is working well and what needs attention, or the results of innovatory practices.

The quality-related information systems required by individual institutions will depend to some extent on local circumstances, but it is at least expected to cover:

• student progression and success rates; • employability of graduates; • students’ satisfaction with their programmes; • effectiveness of teachers; • profile of the student population; • learning resources available and their costs; • the institution’s own key performance indicators.

There is also value in institutions comparing themselves with other similar organisations within the EHEA and beyond. This allows them to extend the range of their self-knowledge and to access possible ways of improving their own performance.

120

1.7 Public information

STANDARD: Institutions should regularly publish up to date, impartial and objective information, both

quantitative and qualitative, about the programmes and awards they are offering.

GUIDELINES: In fulfilment of their public role, higher education institutions have a responsibility to

provide information about the programmes they are offering, the intended learning outcomes of these, the qualifications they award, the teaching, learning and assessment procedures used, and the learning opportunities available to their students. Published information might also include the views and employment destinations of past students and the profile of the current student population. This information should be accurate, impartial, objective and readily accessible and should not be used simply as a marketing opportunity. The institution should verify that it meets its own expectations in respect of impartiality and objectivity.