Quality Audit in the Nordic Countries

8/7/2019 Quality Audit in the Nordic Countries

1/48

QUALITY AUDIT IN THENORDIC COUNTRIES

Report prepared or the Nordic Quality Assurance

Network in Higher Education (NOQA)

Sta an Wahln (ed.)

June 2007


2/48

Nordic Quality Assurance Network in Higher Education (NOQA) www.noqa.net

QUALITY AUDIT IN THE NORDIC COUNTRIESReport prepared or the Nordic Quality Assurance Network in Higher Education (NOQA)

Contents: Stafan Wahln , Te Swedish National Agencys Evaluation DepartmentDesign: Te Swedish National Agencys In ormation Department, Stockholm, June
http://%20http//www.hsv.sehttp://%20http//www.hsv.se


3/48

Contents

Summary 7

Background 9What is quality audit? 9

The Agencies and their tasks 11EVA 11FINHEEC 1Iceland 1NOKUT 13HSV 13

Criteria, principles and ormal consequences o audit 15Formal consequences 1Remarks 18

Methodology 19The experts 19Sel -evaluation 3Site visit 6Language 8Remarks 9

Findings and efects 31Denmark 3Finland 33Iceland 35Norway 35Sweden 3

Strengths, challenges and developments 39General observations 39Observations on elements o the methodology 4Final thoughts 41

Re erences 43

Appendices 45Appendix 1 45Appendix 48


4/48


5/48

Introduction

For well over a decade now, the national quality assurance agencies in the veNordic countries, Denmark, Finland, Iceland, Norway and Sweden, have metannually to exchange in ormation and experiences. A ormalised network, theNordic Quality Assurance Network in Higher Education (NOQA), was estab-lished in 2003. One o the major tasks is to pursue, each year, a joint project o common concern on an aspect o quality assurance o higher education, result-ing in a report published on the NOQA website . Tis year we have taken alook at quality audit in the Nordic countries in order to compare and analyse

the role o audit in the national quality assurance systems, methodologies andndings and e ects, where this has been possible.Te project has been led by the Swedish National Agency or Higher Edu-

cation (HSV) and carried out by a working group consisting o :Irne Hggstrm, Sta an Wahln (project leaders) and Lisa Jmtsved Lund-

mark, HSV, SwedenInge Enroth, Te Danish Evaluation Institute, Denmark Kirsi Mustonen, and Solveig Corner, the Finnish Higher Education Eva-

luation Council, FinlandEinar Hreinsson, the Ministry o Education, Science and Culture, Ice-

land Wenche Froestad, Norwegian Agency or Quality Assurance in Education,

Norway.

Te project started in October 2006 and has generated ve country reportson audit processes. Tey have served as input or the project and provided thebasic in ormation or this report. Some o them are available on the websi-tes o the individual agencies. Te project group has met on ve occasions todiscuss the project and successive versions o the report. Comments rom theannual network meeting in Stockholm in May 2007 have also been taken into

account or this nal version.

Stockholm June 2007

Sta an Wahln

. www.noqa.net www.noqa.net


6/48


7/48

Summary

Quality audit o higher education institutions may be de ned as a processor checking that procedures are in place to in higher education institutions

to assure and improve quality, integrity or standards o provision and outco-mes.

Te role o audit in the Nordic national quality assurance systems has variedover the years. In Sweden it has been reintroduced, and is now one o our or

ve di erent methods, estimated to account or about one-third o the totalevaluation budget. In Norway and Finland it is now playing a major role inconjunction with an accreditation system, and may be seen as the chie met-

hod o quality assurance. Denmark has introduced the method quite recently.Iceland is now establishing a system o accreditation which includes audit.Te use o audit or the purpose o control as in Iceland, Norway, and to

a slightly lesser extent in Finland and Sweden resembles the development inseveral other countries in Europe. Like in Denmark, where audit is moreenhancement-oriented, it very distinctly devolves the responsibility or qua-lity to the institutions, but makes it mandatory or institutions to have a un-ctioning quality assurance system and sanctions may even be imposed i it isnot acceptable.

In all ve Nordic countries it is the quality assurance system as such and

its implementation that are the object o evaluation, i.e. objectives, documen-tation, evaluation and general acceptance and participation in quality work,although the Sweden now also stresses the documentation o output, outcomesand impact o systematic quality work.

Audit is only one o several diferent evaluation methods. Te many approaches may be seen as a burden on both higher education institutions andquality assurance agencies. However, the di erent methods also supplementeach other in various ways. In Norway auditors may discover either weaknes-ses or strengths that may warrant urther investigation and result in revisiono accreditation. Similar developments may be seen in the other countries.

Te general audit methodology in the Nordic countries is based on theprinciples o the European Standards and Guidelines and there are no majordi erences among the countries. A sel -evaluation conducted by the institu-tion is ollowed by a site visit o external experts who prepare a report or theagency in question. In Norway and Finland the sel -evaluation process andreport are replaced by existing material (annual reports etc.) on institutionalquality work. In some o the countries, the agency makes a decision on thebasis o the report, which may result in accreditation or a re-audit a ter a cer-tain amount o time. Judgements are based on pre-de ned criteria relating toquality work.


8/48

Expert panels consist o academics with experience o leadership in highereducation and sometimes also subject area specialists. International presenceon panels is important. All countries have Nordic experts and some alsorecruit experts rom other countries. Te stakeholder and student perspectivesare emphasised through their presence on panels and as interviewees.

Te ve countries are at di erent stages o development o institutional qua-lity work and o audit as a method o evaluation. In Denmark quality work has not yet become a tool or continuous improvement and auditors empha-sise the role o managements to lead urther development. In Finland thereis agreement that the audit process has helped to improve quality assuranceo basic operations. In Norway, too, the exercise has been ound use ul or

urther development, but auditors have ound de ciencies in the ollow-up o internal evaluations and eedback to students. Broad participation in quality

work is another area or improvement. Sweden reports that leadership andorganisation o quality work have developed, as well as policy and strategy

ormulation and ollow-up, but that the quality cycle (planning implemen-ting ollow-up evaluating new planning) as a strategic instrument wasundeveloped in many cases.

Finally it is remarked that external quality assurance is important but thatthe costs and the e orts involved or institutions and quality assurance agen-cies must be reasonable and that the main responsibility or the quality o provision must always stay with the provider. Audit is one method that takesthese two demands into account.


9/48

Background

Among the many things initiated as a result o the Bologna process is theEuropean harmonisation o quality assurance in higher education. One o the main principles o this process is that higher education institutions aresel -regulating, autonomous units, accountable or the quality o their ownprovision o teaching and research. In this context, both internal and externalquality assurance has been seen as important elements in the development o the European Higher Education Area. Tus, as part o the Bologna process theEuropean Association or Quality Assurance in Higher Education (ENQA),in cooperation with the European University Association (EUA), the Euro-

pean Association o Institutions in Higher Education (EURASHE) and theNational Unions o Students in Europe (ESIB) was commissioned, at the 2003meeting o European Ministers o Education in Berlin, to develop standardsand guidelines or quality assurance o both institutions and quality assuranceagencies. Such a documentwas published in 2005 and approved by the minis-ters at their meeting in Bergen in the same year. It is a living document, likely to be revised over the next ew years, but it nevertheless established a numbero important points upon which both institutions and agencies agree.

Some o these points re er to the responsibility o institutions both or qua-lity and or quality assurance and that external evaluation should take this

into account. As a consequence it is natural that quality assurance agenciesshould see institutional audit as a method o quality assurance. A growingnumber have introduced or reintroduced quality audit, among them all the

ve Nordic countries.

What is quality audit?Quality audit in the context o higher education may be de ned as a proc-ess or checking that procedures are in place to assure and improve quality,integrity or standards o provision and outcomes. It may apply to all levels

o higher education institutions, rom subjects, departments and aculties toinstitutions. In the context o this report, we use the term to re er to evalua-tion o institutional systems o quality assurance and enhancement.

Generally, the Nordic quality assurance agencies concur with the abovede nition, as ar as the objective o evaluation is concerned. Tey all agreethat systems should be in place in institutions or the assurance and develop-ment o quality.

. ENQA ( ).ENQA ( ).

. An attempt to interpret and problematise the Standards and Guidelines was a Nordic An attempt to interpret and problematise the Standards and Guidelines was a Nordicproject resulting in a report: ENQA ( ).

. De nition rom Harvey ( ).De nition rom Harvey ( ).


10/48

Te Nordic agencies do not impose a particular quality system but the onein place should be t or purpose and be both e cient and e ective. However,a quality system should meet certain requirements: For example, it should becapable o revealing poor quality; it should contain routines or setting goals,

or evaluation at various levels and ollow-up o results o evaluations; it shouldcontain routines or establishing new provision and continuous assurance andimprovement o existing provision. Te purpose o audits is then to providean assessment o the strengths and weaknesses o such systems and providerecommendations or improvement.

Each country has, however, its own angle, which refects its particular emp-hasis and purpose. For example, there is a diference between the Danishapproach stressing enhancement, and those o Finland, Iceland, Norway andSweden ( rom 2007) putting more emphasis on control and in Norway alsoto accreditation o subjects and programmes and in Iceland to accreditationo elds o study.

Finally, the Nordic agencies also assume that institutional quality work should ideally ollow the quality cycle, i.e. is a continuous circular processbeginning with stating objectives and plans, ollowed by implementation, ana-lysis and revision, leading to the establishment o new objectives and plans,etc.


11/48

The Agencies and their tasks

Tis chapter presents the Nordic agencies participating in the project andthe role played by audit in their quality assurance systems. In all the systemsquality audit is one o several quality assurance methods used alongside, orin conjunction with, programme and subject assessment and other orms o evaluation. It may also in orm, in various ways, the accreditation o subjects,programmes, aculties and institutions

EVA

Te Danish Evaluation Institute (EVA)

is an independent institution estab-lished by the government. Since 2000 it has had the task to evaluate the wholeeducational system in Denmark, not only higher education. In the area o higher education EVA also conducts accreditation o programmes, e.g. voca-tional bachelor, and institutions, e.g. university colleges.

Another core unction o EVA is to develop methods and tools or quality assurance and development and to disseminate knowledge and in ormationon the subject. Finally, EVA conducts a range o income generating activities,such as benchmarking, accreditation, evaluation and development projects orinstitutions and authorities.

A ter a long experience o mainly programme and subject assessment, star-ting in 1992, EVA introduced auditing in 2003, along with continued assess-ment o programmes and subjects, and between 2003 and 2005 conducted ouraudit projects mainly based on a tness- or purpose approach. In 2006 a new audit concept based on pre-de ned criteria was developed. An agreement wasreached between EVA and the Danish universities that all universities shouldbe audited within a given period. However, the conditions or EVAs activitiesin the university area changed in 2007, and a ter recent re orms the Instituteno longer holds the mandate nor receives the unding to initiate projects inthe university area. Current audit processes have, there ore, been interrupted.

Tus, rom now on, EVA will o er to conduct audits at the universities ownexpense. However, in the area o university colleges, where EVAs position hasnot changed, and the Institute is presently developing a concept or institu-tional audit or these institutions.

. www.eva.dk www.eva.dk


12/48

FINHEECTe Finnish Higher Education Evaluation Council (FINHEEC) , establishedin 1996, is an independent expert body o the Ministry o Education assisting

universities, polytechnics, and the Ministry in matters relating to evaluation.FINHEEC evaluations all into our categories: 1) institutional evaluations(e.g. the audits); 2) programme evaluations; 3) evaluations relating to nationalhigher education policy objectives and other thematic evaluations; and 4) eval-uation and registration (accreditation) o pro essional courses o ered by highereducation institutions.

In addition to these, the Ministry o Education commissions evaluationsbe ore designating Centres o Excellence in Education. FINHEEC also pro-vides ee-charging services, or example evaluations o institutions that are notoperating under the Ministry o Education.

In its operations, FINHEEC has always emphasised the principle o enhan-cement-led evaluation. Tis means that the evaluations produce in ormationabout higher education and its quality which can be used in institutionaldevelopment. Tis in ormation is also used by the Ministry o Education, orexample, in per ormance-steering and decision-making.

Since 2004 FINHEEC has been developing an audit procedure ocusingon the quality assurance systems o higher education institutions. Te aim isthat in 2011 all the quality assurance systems o the Finnish institutions willhave been audited.

IcelandQuality assurance o higher education in Iceland is the responsibility o theMinistry o Education, Science and Culture. A special unit within the Minis-try has been responsible or the evaluation o institutions. Te Ministry recru-its national and international experts to conduct audits o departments orinstitutions.

In 2006, a new law has come into e ect, introducing accreditation o allelds o study in higher education institutions. Initial accreditation o HEIs

will be ollowed up by external evaluations, which include a quality audit.

Tus, a ter an initial accreditation exercise, institutions can apply or anaccreditation o PhD programmes in elds o study that have already beengranted accreditation or undergraduate and masters programmes. From 2008onwards departments, whole institutions or even whole disciplines at Icelandichigher education institutions will be audited in cycles in accordance with athree-year plan, set by the Minister o Education. Te objective o the auditthat will ollow the accreditation is two old. First, its goal is to make sure that

. www. nheec. www. nheec.

. www.mrn.stjr.is www.mrn.stjr.is


13/48

the basis or accreditation is still in place, and second, the audit has an aspecto enhancement.

NOKUTTe Norwegian Agency or Quality Assurance in Education (NOKU ) isan independent public agency, established by law in 2003, with the task o carrying out external quality assurance o Norwegian higher education andtertiary vocational education by means o accreditation and evaluation. As theNorwegian ENIC-NARIC centre NOKU also considers applications orgeneral recognition o oreign higher education quali cations.

Besides audit, NOKU s evaluation responsibilities include accreditationo institutions, accreditation o course provision, revision o previously granted

accreditation and other evaluations to investigate the quality o Norwegianhigher education including assessment o the quality o various disciplines orprogrammes. Te last-mentioned activity may be carried out at the behest o the Ministry o Education. Audit, de ned as evaluation o the institutions sys-tems or quality assurance, is central in that it a ects all institutions providinghigher education. Finally, NOKU administers the Ministry o Educationsprize avarded annually or outstanding quality work in higher education.

HSV

Te Swedish National Agency or Higher Education (HSV) is a govern-ment agency established in 1995, entrusted with evaluation o higher educa-tion, higher education statistics, legal supervision, in ormation to students andprospective students, research and policy advice and recognition o oreignquali cations.

Te role o institutional quality audit in reviews carried out by HSV hasvaried over the years, since the introduction o national quality assurance in1995. It was then the main orm, but simultaneously, assessment o subjectsand programmes o special interest took place as well as accreditation o pro-

essional programmes and masters programmes. From 2001, a ter two cycles,

audits were replaced as the major orm o national quality assurance by anextensive six-year programme o subject and programme assessment coveringall provision leading up to a degree including Ph.D.

From 2007, there will be a new audit cycle, which will run parallel to a simp-li ed subject and programme cycle. Accreditation o new pro essional pro-grammes at all higher education institutions and o the new Bologna Masters

. www.nokut.no www.nokut.no. Te ENIC Network (European Network o In ormation Centres) and the NARIC Network Te ENIC Network (European Network o In ormation Centres) and the NARIC Network

(National Academic Recognition In ormation Centres) deal with question o recognition o academic studies and degrees.

. www.hsv.se www.hsv.se


14/48

degree at university colleges will also take place, and a prize or excellencein higher education will be introduced. A pilot audit o a university college(Sdertrn University College) is now in its nal stages and will contribute to

urther methodological re nement.


15/48

Criteria, principles and ormalconsequences o audit

Te Standards and Guidelines for Quality Assurance in the European Higher Education Area(ENQA 2005, hence orth re erred to asthe European Standards and Guidelines ) state that any ormal decisions made as a result o an externalquality assurance activity should be based on explicit published criteria thatare applied consistently.

Te use o the word criterionvaries in the descriptions o the di erent Nor-dic countries. In Denmark and Norway it is used mainly in the sense o re -erence points that invite refection, discussion and commentary , but which

should be met, at least at a basic level, or the quality system to be acceptable.Tese re erence points are termed auditing targets in Finland and aspectsin Sweden. Te term criterion in the latter two countries is reserved or express-ing a particular level o per ormance that should be met. In this report theterm criterion is used as in Denmark and Norway. It is expected that internalquality systems should live up to what is expressed by the criteria.

All the Nordic agencies are working towards bringing or have already brought their audit concepts into agreement with the European Standardsand Guidelines. Tey are thus , explicitly or implicitly, incorporated into theagencies audit criteria and targets.

Te ocus o audits and the criteria in Norway is on the system, its objec-tives, its acceptance among students and sta and the way in which it helpsto develop a quality culture in the entire institution. It is understood that thesystem should include the collection o data and other in ormation rom inter-nal evaluations, which should be analysed and used or decisions on internalresource allocation and prioritisations. Te internal process o sel -evaluationmust include an annual report on quality to the institutions board.

Some o the Danish, Finnish and Swedish criteria are more speci c andre er to particular elements o quality systems. For example, HSVs aspectsinclude a series o quality actors which the other systems do not mentionspeci cally, among them internationalisation, gender equality and social andethnic diversity and among FINHEECs targets and EVAs criteria we ndsta development .

In Iceland the objectives o the quality assurance o both teaching andresearch in the higher education institutions are to ensure that the require-ments or accreditation o the institutions are met, to ensure that the quali ca-

. For lists o criteria used by the Nordic countries, see Appendix .For lists o criteria used by the Nordic countries, see Appendix .

. See ENQA ( )See ENQA ( )

. In the European Standards and Guidelines, part I, assurance o teacher quality is re erredIn the European Standards and Guidelines, part I, assurance o teacher quality is re erredto as one o the important standards.


16/48

tion ramework or higher education and the respective degrees is ul lled, toimprove the quality o teaching and research and to ensure the competitivenesso the institutions at international level.

How are criteria used?

In Denmark the criteria do not express minimum acceptable standards. Rather,they are statements about a mature and well-developed system or quality assu-rance and development. Each criterion should be reported on in the sel -eva-luation report. And in the nal audit report the panel will give its assessmento how the university meets each criterion. However, the assessment and theaudit report will ocus on how and to what extent the universitys quality work

ul ls the demands o the criteria, and not only on the more simplistic i .Te overarching objective o quality audit in Norway, as interpreted on the

basis o the criteria, is to assure and enhance the quality o studies at the highereducation institutions. Te regulations o the Norwegian act on higher educa-tion de ne what should be included in the HEIs quality assurance systems. Itis expected that institutional quality systems should be able to identi y goodand poor quality, and thus provide a basis or positive change. Te evaluationo the institutional quality assurance systems includes both the systems struc-ture, the documentation it produces and the assessments o educational quality conducted by the institution itsel .

In accordance with its regulations NOKU re ers in its audits to 10 cri-teria, prepared in consultation with the higher education sector. However,

the documentation submitted to NOKU and the panel is not meant toanswer explicitly these given criteria. Instead it must give a realistic pictureo the institutions current work on quality and its quality assurance system.However, the reports always analyse the institutional quality assurance sys-tems according to the criteria. Te panel is required to provide clear advice on

whether the quality assurance system as a whole should be approved or not, with regard to NOKU s criteria and the regulations o the Act relating touniversities and colleges.

In Iceland the ocus is on the compliance with the higher education actboth with regard to internal quality systems and, among other things, admi-

nistration and organisation, organisation o teaching and research, sta qua-li cation requirements, admission requirements and students rights and obli-gations. Further, expert panels are to report on expertise and competence in aparticular eld o study, cooperation and support o the university in the eldo research, teaching sta and experts in any particular eld. Also they willcomment on special attention to elds o research and the status o elds o study and subdivisions in national and international comparison.

Te criteria in both the ormer and the coming Swedish models largely resemble those in the other countries, but the new system also attempts tolink quality plans and activities to results o a quality work in the sense o

improved per ormance and long-term e ects (e.g. employer satis action and


17/48

employment rates). Tus, the audit model requires institutions to conductinvestigations and surveys and monitor progress in terms o output, outcomeand impact to be able to demonstrate progress in terms o enhanced quality.Tis is a notoriously problematic area, needing urther development. A directcause e ect relationship between improved quality work and student per-

ormance is hard to prove.Finland and Sweden are quite explicit in determining di erent levels o per-

ormance. In the Finnish model our levels are described or each criterion:absent, emerging, developing and advanced, and criteria are developed or eacho them. Te new Swedish system distinguishes three levels: initiated, underdevelopment and developed, assuming, on the basis o the two previous auditcycles, that the implementation o each aspect has been at least initiated. Inneither o the two countries are the di erent criteria intended as grades tobe awarded or each aspect or target, but rather as indicators to acilitate theexperts discussion.

Formal consequencesIn Nordic audits, the control/assurance element and the enhancement objec-tive are both present. It is not a question o either or, but a matter o degree.Tese perspectives are partly refected in the ormulation o the criteria app-lied, partly also in the ormal consequences, i any, o the audit, or the use to

which it is put.

Tere seems to be a shi t o emphasis, however, which points in the direc-tion o assurance. Tis is refected in the ormal consequences o audit results.Norway and Iceland apply ormal sanctions on those institutions whose qual-ity assurance systems are not up to standard.

Quality audits in Norway result in a judgement o the extent to which thesystem as a whole is satis actory and indicate areas o development. I NOKU

nds undamental de ciencies in the quality assurance system, the institutionis granted a time limit o six months to recti y matters. NOKU will thenconduct a new audit. I the institutional quality assurance system ails approvalagain, the institution will lose its right to start or to apply or approval o new

provision. A ter one year the institution may ask or a new audit. Te auditpanel may also advise NOKU to initiate revision o accreditation already granted. Also, the Board o NOKU may initiate such revision on the basiso the audit report itsel .

In both Sweden and Finland, an incomplete or unsatis actory quality assur-ance system will result in a re-audit, in Finland a ter two years, in Sweden a terone year. Decisions on these matters are taken by FINHEEC in Finland on apass ail basis, and, rom 2007, by the Chancellor o the Swedish Universi-ties (the Head o HSV) in Sweden, who will express either ull con dence,


18/48

con dence or limited con dencein the quality assurance programme andits implementation. Re-audits will concentrate on the improvement measuresproposed by the experts. In neither o the countries have re-audits taken placeyet.

Although there are no ormal consequences in Denmark, and only lim-ited consequences in Finland and Sweden, a negative outcome o an audit isexpected to have an impact on the reputation o an institution, and thus toa ect indirectly e.g. external unding and student intake.

Remarks All countries use prede ned criteria refecting the national quality assurancesystem in place and refecting what is required o institutional quality work.

Tey make these requirements clear and transparent to institutions as well asto experts. Tere is, however, a risk involved in applying the criteria too rigidly on the part o both institutions in their quality work and experts in their jud-gements, and thus to limit creativity and development in quality work. Tere isa clear awareness o this danger, which is discussed urther in the nal chapterStrengths, challenges and developments.

. Te exact wording o these decisions is still provisional.Te exact wording o these decisions is still provisional.


19/48

Methodology

The expertsAcademics

External assessment o research and teaching in higher education has a strongtradition o peer review. Te aim o quality audit is, however, di erent, andthe term peer is not quite adequate. Te peers needed or evaluation do notparticipate on the basis o subject or discipline expertise, but rather on the basiso knowledge and an understanding o quality assurance (o higher education).

An audit panel needs to include academics, but academic leaders and/or aca-demics with a substantial quality assurance record rather than subject specia-lists since the ocus is on quality work and not on the contents and teachingo subjects or programmes and because leadership is an important ingredientin success ul quality assurance development.

Te approach o the Nordic countries with regard to the selection o expertsvaries. Commonly, at least one panel member is recruited rom academicmanagement at institutional or aculty level, i.e. a rector or a dean.

In Iceland relevant academic expertise is a criterion or the selection o experts. Tis is partly due to the act that audits in Iceland are conducted at

eld o study level. Tis is also true in Denmark in cases where audits are car-ried out at aculty level.

NOKU requires that one panel member should have quali cations at thelevel o pro essor, but has not made relevant pro essional knowledge an expli-cit criterion. Occasionally, when the institution to be evaluated is a specialiseduniversity college it has been concerned with this matter, and NOKU tries torecruit audit experts rom di erent academic elds. FINHEEC ensures thatthe academic experts represent di erent higher education sectors (universitiesand polytechnics) and di erent sta groups (management and administration,teaching and research and support services).

Competence rom the eld o quality assurance or evaluation in higher edu-cation is regarded as important. In the Nordic countries at least one membero an audit panel usually has such competences. Tese persons have di erentbackgrounds, such as researchers within the eld o quality assurance, quality assurance managers rom institutions etc.

International comparison is important. Tis requires experts rom abroad.In Iceland, all quality evaluations are done in English with experts recrui-ted rom oreign countries. Tis is to get an international perspective and toensure that a wide pool o experts is available or the task. In Finland andDenmark, institutions may choose between a domestic and an internationalaudit group. Also, in Denmark and Norway there is a tradition o having atleast one expert rom one o the other Nordic countries. For practical (mainly


20/48

language) reasons this is also the case in Sweden, although attempts are madeto nd experts rom other countries who have a working knowledge o a Scan-dinavian language.

Stakeholder, student and other perspectives

One reason or conducting external evaluation is the need or stakeholders,i.e. government, employers, students and the general tax-payer to be in ormedabout the quality o higher education and on whether their money is wellspent. Tis is why it is natural that they should be represented on panels, butalso because they provide alternative, external perspectives on activities.

In all countries except Norway, the external stakeholder perspective hasbeen emphasised, and it has been customary or one expert to come romoutside the academic world. He or she may be a representative o employers

in business and industry or o the community at large (civil service or localgovernment).Students are represented on audit panels. Tose appointed have usually ser-

ved on boards or committees o higher education institutions or other deci-sion-making bodies. In Finland, Iceland, Norway and Sweden where there arenational student unions, the student experts are nominated by these organisa-tions, but appointed by the agencies.

Finally, it should be added that the Nordic countries have a strong tradi-tion o gender equality and it is taken or granted that panels aim at havingan equal representation o men and women.

Selection and appointment o audit experts

In Sweden, Finland and, rom 2006 in Norway, institutions may proposenames o panel members, although it is the prerogative o the agencies toappoint any person who meets the requirements. In all countries the insti-tutions under evaluation will be allowed to comment on the expert panel toensure that there is no confict o interests.

raditionally panels conducting various external evaluations o higher edu-cation are appointed separately or every speci c evaluation. Te audit modelhas, however, raised the question o whether the same experts should con-

duct more than one evaluation. NOKU has the most ar-reaching practiseamongst the Nordic countries in this respect. A group o auditors is appointedannually, and they normally stay with the task or three to our years. FIN-HEEC, appoints some o the experts or two or more audits.

Te rst audits conducted by EVA took their starting-point in the indivi-dual university, its characteristics and challenges and the audit panels com-bined pro les varied accordingly. However, as the Danish audits now ollow

. FINHEEC has a register o the potential audit experts that have been proposed by theFINHEEC has a register o the potential audit experts that have been proposed by the

higher education institutions. However, the institution cannot propose the experts or itsown audit panel.


21/48

prede ned criteria, the experts skills will di er to a lesser degree. Hence eachexpert may conduct several audits.

In the rst two cycles o Swedish audits the experts were appointed sepa-rately or each evaluation, but several o them were, in act, recruited seve-ral times. However, when starting the new cycle o audits, HSV will regardexperts in a certain year as one group regardless o the number o institutionsaudited, and some o them will participate in two audits in one year in orderto make comparison between institutions possible.

The role o project managers

In all countries, project managers are responsible, in some cases with the helpo a project group, or the planning and organisation o the audit, includingthe recruitment o experts. With the exception o Iceland, project managers

belong to the sta o the agencies.Te experts, who are ormally appointed by the boards (or in Iceland theMinistry), are responsible or reports and commonly play a major role in theircreation. But normally it is the project managers that unction as secretaries. InDenmark, project managers are assisted in this task by students on temporary assignments, who take notes during the site visits and thus release the projectmanager rom purely administrative chores, making it possible or him/her toassume a more active role.

With time, project managers acquire considerable knowledge and expe-rience o quality assurance and uno cially they may adopt, at least partially,

the role o expert. Tis is a development that needs to be urther discussed,and which will have an impact both on how reviews are prepared and imple-mented.

Preparing the experts or the task

Experts are prepared or their task in several ways. Tey need to be in ormedabout the context o evaluation, the speci c audit methodology, their role asauditors and ethical concerns. Internationally recruited team members andstakeholders will also o ten require a brie ng on the higher education systemand the relevant legislation in the country.

Generally speaking, introduction sessions are airly brie , and in Norway, where auditors are asked to be available or several years, there is little need

or a thorough grinding. However, initially NOKU arranged two seminarsor its sta and the auditors, and continues to provide one seminar each year,

in which they are updated on any new developments and experiences romaudits conducted. Any newly recruited experts will be brie ed more in detail.

EVA arranges a one-day panel meeting at which EVAs project group acqu-aints panel members with the audit method and its objectives as well as theroles and responsibilities o the panel and EVA, Special attention is paid tothe site visit and the interview techniques to be used. Te Danish higher edu-


22/48

cation system and the relevant legislation are introduced to guide the oreignpanel members.

FINHEEC arranges training seminars over a day and a hal , ocusing onamong other things: the responsibilities o the audit panel, the objectives,methods and criteria o audits, and the experiences o ormer auditors, audi-ting techniques and questions, analysis o audit materials and reporting and,

nally, a discussion o the audit criteria based on team work.In Sweden, a one-day seminar has been organised, concentrating on pre-

vious audit experiences and material (sel -evaluation reports, audit reports) toillustrate methods and approach. A similar process is oreseen or the comingsix-year cycle.

In Iceland, the members o the review team do not get any training due tothe act that Iceland only assigns oreign experts with a wide range o trainingin other countries.

The experts roles and mandates

Te chairs o the panels have special obligations besides chairing meetings.Tese include the preparation o the site visit, in some cases a pre-meeting withthe management o the institution, and in most countries overseeing the pro-cess o writing the audit report. However, the audit experts are all responsible

or the outcome, and naturally they are ully responsible or the conclusionsand advice included in the reports.

Te experts carry out their tasks within national models o external qua-

lity assurance and their work is regulated by mandates given by the agencies.Hence, the Nordic countries vary in respect o the expectations and the rangeo (academic) reedom o the audit experts.

In all countries the audit model de nes the in ormation and documenta-tion the audit panels get rom the institutions to be evaluated. However, thepanels may request additional in ormation when needed. Tere are prede nedprocedures with regard to the groups an audit panel must meet, but the panelmay decide to meet with other groups or units as well. ypically, the agency and the audit panel work together to prepare the site visit.

In Finland the audit panels take an active part in the writing o the audit

reports. Te panel members divide the criteria o the audit among them andgather observations and evidence during the site visit. A ter the site visit thepanel members write dra ts or the report text based on their respective cri-teria at the visit and also their proposals or conclusions. Tese texts are thebasis o the FINHEEC project managers rst dra t o the audit report. Teother Nordic countries provide the audit panel with a secretary (the projectmanager, or, in Iceland, a specially recruited project secretary) who writesdra ts or the report on the basis o the audit panels discussions and notes. Inall countries the audit panels submit comments on dra t versions be ore thereport is nalised.


23/48

Feedback on processes and results is seen as important and or this reasoneedback con erences are commonly arranged at which expertsand represen-

tatives o institutions participate.Tus, FINHEEC arranges a publication con erence at the institution a ter

the board has made its nal decision with regard to the audit. Te chair o theaudit panel presents the results, and the representatives o FINHEEC (chair,secretary general and project manager/s) also take part and present the n-dings. NOKU has annual con erences with the tertiary education sector, to

which the audit experts are invited. Te audit experts are requently asked tocontribute to seminars at the con erence.

Sel -evaluation

The sel -evaluation conceptSel -evaluation and the preparation o a sel -evaluation report are commonprocedures in academic evaluation, in some o the countries required by law.Tey give the unit under review the chance o looking at their quality systemsand assessing what works and what could be improved. Tey also provide asound basis or the work o the expert teams.

Sel -evaluation is part o the Danish, Icelandic and Swedish audit process.In Finland and Norway, however, it is replaced by providing existing material

rom the institutions own quality systems. One reason or this is that the man-datory quality assurance systems should produce annual reports on the quality

o provision and the institutions work on quality. Also using the documentsthat have been produced in the course o regular processes is considered to givea more realistic picture than a sel -evaluation report made speci cally or theaudit. It puts less o a burden on universities and colleges under review, whileat the same time requiring them to build up and develop regular systems whichgenerate relevant material on the quality o their provision.

Te period or sel -evaluation (or similar initiatives, herea ter all re erredto as sel -evaluation) is rom 3 to 4 months. At deadline the institution mustsubmit the report and other material to the agency which then orwards it tothe panel members. In Norway, the panel and NOKU normally also getintra-web access to the institutions quality assurance system and data. Examp-les o quality reports rom di erent organisational levels are required in order

or the panel to be able to ollow the audit trail rom the institutional level tothe aculty, the department and even to an individual study programme or aspeci c thematic course, in order to clari y the analysis o strengths and chal-lenges and the measures taken to meet them.

Te deadline or submission is rom our (Finland) to twelve (Iceland) weeks be ore the site visit. Te length o the Icelandic time span is due to the

act that the sel -evaluation has to be translated into English.


24/48

Guidance to the higher education institutions

All the Nordic quality assurance organisations provide some kind o guidanceor guidelines to the institutions in connection with the sel -evaluation.

FINHEEC provides a comprehensive audit manual explaining quality assurance and the auditing concept and process and presenting the criteriaor assessment. EVA provides institutions with a slightly less comprehensive

document on the same subjects. Tis document is sent to the institution underreview and shortly a ter presented at a launch seminar at the university withthe participation o the top management o the university and EVA and with asmany participants rom the university as possible. Te seminar is expected tocall attention to the sel -evaluation process in particular and to the importanceo quality work in general and is also an opportunity to give in ormation andclari cation with regard to the audit project and thus meet or anticipate pos-

sible criticism.In Iceland the institutions will be provided with guidelines rom theMinistry o Education regarding the content and orm o their sel -evalua-tion report.

Representatives o agencies meet with the institutions early on in the pro-cess, and in Sweden and Finland the chair o the panel participates as well. Tepurpose is both to clari y the task and agree on special considerations.

Te sel -evaluation reports vary in length rom 25 30 (Sweden) to 80 100pages (Denmark). Iceland requires that they should include a summary o themain ndings. In addition to the report proper all countries allow or appen-

dices, and o ten more is submitted than is needed. It may be added that whensel -evaluation proper is not required, it is more difcult to anticipate and limitthe amount o documentation to be submitted.

In all countries the audit panel or the agency may ask the HEI or additio-nal in ormation be ore the site visit.

The contents o the sel -evaluation report

Whether institutions are expected to submit sel -evaluation reports proper orother already existing material, the in ormation required by the agencies isquite similar. It should be both descriptive and analytical and provide evidence

o how quality work is implemented and linked to institutional strategy. Ide-ally, it should give an account o the results quality work produces in variousdimensions, address challenges that have been identi ed and suggest ways

orward. Criteria or assessment are to be ound in Appendix 1.EVA asks the university to prepare a sel -evaluation report written as an

independent text. It should provide in ormation on quality work or all aspectsrelated to the education area (and research i included). It should contain ade-quate descriptions and nuanced refections on the quality work as it is organi-sed and planned, but just as important, how it is carried out in practice. Temain points rom central documents may be outlined in the report where rele-

vant, and the ull documents should be enclosed or substantiation.


25/48

FINHEEC asks or material that provides a su cient in ormation base orthe auditors to assess the comprehensiveness, per ormance, transparency ande ectiveness o the quality assurance system. Te audit material must also out-line the institutions organisation, describe the structure o the quality assu-rance system and its links with the management system and provide evidenceo the per ormance o the system. Institutions are asked to write a short sel -evaluation o the strengths and weaknesses o their quality assurance systemas a part o their audit material.

Te institution is also expected to submit material to substantiate the per-ormance o its quality assurance system, by providing evidence concerning

each o the ten auditing targets. Te material should indicate clearly whichevidence relates to which target.

NOKU expects the in ormation and documentation generated by theinstitutions quality assurance system to have been analysed in an annualreport to the institutions board. Tis in ormation and the institutions use o it in its decision-making procedures orm central elements in the audit panelsanalysis. NOKU asks or documentation which includes a detailed descrip-tion o the quality assurance system, internal reports on quality, key- gures oneducational quality used by the institution and important steering documents.

Also, the documentation must show how the institutions work on educationalquality is organised to ensure broad participation rom students and sta andhow it relates to the institution strategy.

In Sweden, HSV has up until now asked or sel -evaluation reports provi-

ding an account and analysis o processes designed to assure and improve thequality o all provision by the institution in relation to a number o aspects.Tese aspects were assumed to characterise good quality work in all institu-tions.

Te coming audits will be more stringent with regard to aspects and crite-ria or assessment (in accordance with the European standards and guideli-nes or quality work). Tey will also ask or documentation o results in the

orm o output, outcome and impact. Furthermore, other material, such askey per ormance indicators and relevant extracts rom other assessments andinvestigations will be placed at the disposal o the experts. Te departments

or other units selected or interviews will be asked to submit existing materialto substantiate their quality work and its relationship to the quality process o the institutions as a whole, but not to submit sel -evaluation reports.

Te Icelandic sel -evaluation reports are expected to contain a reasonedassessment o the strengths and weaknesses o the study programmes, basedupon precise data where applicable. Guidelines rom the Ministry o Educa-tion include de nitions o the statistical data required. Most o the items lis-ted in the checklist are sel -explanatory. But it should be stated that the qua-lity o the study programme should always be evaluated in view o the overallpolicy o the relevant higher education institution, and the manner in which

this policy is implemented in the policy o the relevant institution or aculty.


26/48

Te organisation o study, study materials and courses should refect and servethese objectives.

For each item o the sel -evaluation report, a description and analysis o strengths and weaknesses are required, together with observations on how theinstitution/ aculty intends to resolve the problems/ aults pointed out. It shouldbe pointed out that statements made in the sel -evaluation do not have to beproved; it is su cient to give examples and explain. More detailed data may be placed in appendices.

Site visitAim/purpose

Te aims o the site visit resemble one another. It is a question o providing

a clear and detailed picture o the institutional quality work and/or quality assurance system by supplementing the in ormation and analysis given in thesel -evaluations and/or other documentation. Te site visit is intended to be asan interactive event contributing to the development o the quality assurancesystem and, i possible to nd examples o good practice to disseminate toother universities and colleges.

Implementation

Te site visits include dialogues between the panel and various groups in theinstitutions under review. Te eedback rom the evaluated institutions o tenpoints out that these meetings and dialogues are constructive and use ul.

In all the Nordic countries the duration o the site visit is 25 days. It is nor-mally quite a hectic event, which consists o a number o meetings and exten-sive interviews (Appendix 2). Te total number o the persons interviewed atone single site visit may be as large as 200. Te number o persons varies withthe length o the visit and the size and complexity o the institution or aculty.Be ore the visit the higher education institution will receive instructions romthe agency how to put together the di erent groups to be interviewed, how many persons the panel wants to meet in each group and any other elementsthat the site visit schedule should include. Te visits usually begin and end

with a meeting with the representatives o the leadership and managemento the institution. Also the representatives (e.g. the teaching and adminis-trative sta ) o various units within the higher education institution and itsdepartments are interviewed. Some o the most important meetings are withstudents. O ten, though not in all the countries, panels also meet externalstakeholders, e.g. business and working li e representatives, i applicable, andexternal members o the institutional boards and graduates in order to collectviews on the relevance o the institutions work on quality.

ime permitting, various acilities such as lecture theatres, laboratories andlibraries may also be visited.


27/48

At the end o the visit the Finnish, Icelandic, Norwegian and Swedishpanels usually present their preliminary impressions to the leadership o theinstitution. Te Danish panels dont give any assessments be ore their report,i.e. until they have gathered and analysed all the in ormation and documen-tation. Te project managers rom all the Nordic agencies (and in Denmark also the student worker) take part in the site visits and take notes at the mee-tings as secretaries o the panel.

Institutions choose their representatives or the site visit meetings on thebasis o the instructions, the audit visit programme and any other criteria esta-blished by the agency and the audit panel. However, the panel also reservesthe right to meet any other persons or groups. Student interviewees are bothstudent union representatives and ordinary students.

In Finland, Norway and Sweden, i need be, the auditors may break up intosmaller groups in order to be able to get as ull a picture o the institution aspossible. Te disadvantage o such an approach is that not all experts get thesame picture o the institution, which may make it problematic to draw thecorrect conclusions.

It is important that all members o the expert team take an active part. TeDanish reliance on students or note taking makes it possible or the projectmanager also to participate in the discussions. Student panel members may have a particular task to speak up on questions o student a airs, but their roleis not restricted to that theme.

Audit questions

Practices or producing the audit questions or the site visit vary.Te sel -evaluation report or other documentation rom the university orms

the basis or the site visit and the questions posed at the site visit. EVAs projectgroup prepares a dra t question guide which is adapted and approved by thepanel in advance. In Finland, Norway and Sweden the panels ormulate theaudit questions or each audit.

In Finland the audit questions are prepared on the basis o auditing targetsand criteria. Te panel members divide the writing responsibilities be ore thesite visit and they also produce questions on their own audit targets be ore

the visit. Te list o questions will be completed by the project manager romFINHEEC and checked by the panel be ore the site visit.

The reportIn all the Nordic countries the quality audit like all national evaluations alwaysresults in a public report. In Iceland, the report largely ollows the outline andthe structure o the sel -evaluation report. It is also common practice that thereport should ollow a relatively uni orm structure and relate to criteria set upby the agency in each o the Nordic countries.


28/48

Te structure o the report usually involves a description o the audit pro-cess, a description o the institution and its quality assurance system and itsimplementation and nally the result o the audit. Te enhancement aspectrequires recommendations on how the quality system should be improved.Te report should, there ore, ocus and elaborate both on how and to whatextent the quality work is acceptable, and on what improvement is desirableor needed.

Although the reports are made public in the Nordic countries, o ten witha press release and press coverage, at least rom the local media, it is commonpractice, and in Denmark required by law, or the higher education institu-tion in question to get a chance to examine a dra t o the report to check or

actual errors and misunderstandings. Neither in Finland, Iceland, Norway nor Sweden does this dra t include the panels nal conclusion.

Te published reports in Finland, Norway and, rom 2007, in Swedeninclude a decision taken by relevant agency as to the acceptability o the quality system and its implementation. In Finnish reports, there are both the proposalo the experts and the decision o FINHEEC with regard to the acceptability o the quality assurance system. Tese two may di er. In Norway and Iceland,also the audit panels report to the agency (Ministry) must contain a decisionon whether the institutions quality system is approved. In almost all cases the

Agency (Ministry) will ollow the panels decision. Finally, Norwegian reportsinclude a ormal statement on the report by the evaluated institution.

LanguageOne o the advantages o evaluation is to be exposed to not only internal butalso external views on ones activities. Tis is a reason why, or example, theStandards and Guidelines insist on the importance o stakeholder and inter-national presence on evaluation panels.

International participation in quality reviews o ten imposes certain require-ments on the implementation o various phases o the evaluation. For exam-ple, oreign experts need to be brie ed thoroughly not only regarding evalu-ation methods but also about the (higher) education system in the country.

Te question o language will also have to be taken into account at variousstages o the process. In the vast majority o cases the choice is between thenative language(s) and English. Tis has been solved in the Nordic countriesin di erent ways.

In Norway and Sweden evaluations, including audits, are carried out in thenative language. In Finland and Denmark the situation varies. Institutionsare ree to choose language, i.e. whether they want the audit to be conducted inEnglish or in the native languages. Icelandic audits are carried out exclusively

. In Finland Finnish-speaking higher education institutions mostly use Finnish and Swedish-In Finland Finnish-speaking higher education institutions mostly use Finnish and Swedish-speaking institutions have so ar chosen Swedish.


29/48

in English due to the act that the Icelandic academic community is small butalso because an international perspective is regarded as necessary.

Te choice o language has repercussions or all the stages o the audit. I English is chosen, the institutions will have to submit sel -evaluations in Eng-lish and other relevant material will have to be translated.

For the agencies the use o English is probably less o a burden. It means thatin ormation material, including relevant parts o higher education regulations,

will have to be available in English. Also the nal report will be in English. Forall those involved in interviews English will be the language o discussions.

Conducting audits in a oreign language may lead to misunderstandingsand loss o in ormation. Te vocabulary used may not be amiliar to thoseinvolved, including students and external stakeholders who read the report.On the other hand, views rom the outside world are essential or benchmark-ing, and the Nordic experience is, on the whole, positive in those cases whereEnglish has been used.

RemarksTe audit methodologies do not vary substantially. Te European Standardsand Guidelines have had a clear infuence on the way audits are conducted.

Experts are selected on similar grounds, the main exception being Iceland, where audits are linked closely to evaluation o elds o study. Iceland is alsothe only one o the countries where international (also non-Nordic) panel

members are a standard eature, whereas the other countries mostly rely onthe participation o Nordic experts.Te main di erence concerns the sel -evaluation process and report, which

are requirements in three o the countries, whereas Finland and Norway rely on existing material. Te pros and cons o these procedures are discussed inthe nal chapter.


30/48


31/48

Findings and e ects

Tis chapter discusses three areas: How have institutions perceived the legaldemands or quality work? How have audits been received by auditors andinstitutions? And what have been the e ects o audits? Te accounts vary, butare based on gathering o data, except or Iceland, where there is yet limitedexperience o the audit model in the new quality assurance system. One gene-ral and very obvious conclusion is that the implementation and general accep-tance o systematic quality work in higher education institutions takes time.

Te ve countries are at diferent stages o development o both quality work as a development strategy in institutions and the development o audit

as a method o evaluation. Denmark has recently taken up the method. Tesame goes or Iceland. Audits have demonstrated that in Denmark institutionshave not yet ully developed systematic quality work as a tool or continuousimprovement. Further, auditors stress the role o managements to lead thedevelopment o quality work and clari y the advantages.

In Finland eedback collected rom both auditors and institutions a ter tenaudits, shows a positive picture, at least o the administration and implemen-tation o the audits. Tus, there is general contentment on the part o the insti-tutions with regard to in ormation, audit material, the audit visit, criteria, andthe reports. Tere is agreement that the process has helped to improve quality

assurance o basic operations. Audit teams appreciated the model includingthe division o work within the team and use o the criteria and the sharedresponsibilities or writing. Tere were, however, certain reservations about thesize o the material submitted by the institutions. A conclusion drawn also inFinland is that the role o management is crucial to the acceptance and suc-cess o institutional quality assurance.

In Norway an analysis o institutional statements on audit reports indicatesthat almost all audited institutions nd the exercise use ul or urther deve-lopment. Only three out o 35 institutions raise a discussion on the principleso the audit or the validity o the methodology. Institutions have developed

quality assurance systems. Studies on the reports show, however, that there isroom or improvement. For example, annual reports on educational quality do not orm a su cient basis or decision-making; there are de ciencies in the

ollow-up o evaluations, including student questionnaires (particularly courseevaluations) and there is insu cient eedback to students. It is also di cultto achieve broad participation on the part o students and teachers. It is theimpression, however, that students know how to use the quality assurancesystem with the help o their representatives or those responsible or quality assurance at the institutions, whenever necessary.

A study o the impact o the rst two cycles o audits in Sweden shows that

they have probably given rise to developments in leadership and organisation


32/48

o internal quality work. Te same goes, to a slightly lesser extent, or e.g.student infuence, policy and strategy evaluation and ollow-up. It is worthnoting, however, that the implementation o the quality cycle (setting objecti-ves implementing evaluating setting new objectives) had been achievedin only a minority o the institutions.

DenmarkTe completed audits have shown that universities are generally making nume-rous and serious QA e orts, but that there continue to be challenges. It seemsthe main common challenges concern ensuring coherence and a systematicapproach to quality work, as well as seeing the possibilities in using it to theuniversitys own advantage.

Te audits show that quality work o ten seems abstract and isolated compa-red to academic core activities. Among the scienti c sta it is o ten regarded assomething that takes valuable time rom real (read: academic) work. In generalthere are strong ambitions to o er high-quality provision, but not everyonethinks that quality work is the way to do this. Tus, it is an important task

or the universities to discuss and identi y how the academic milieus and theindividual teacher and researcher can bene t rom it.

Furthermore, the audits have revealed that quality work is o ten establishedto meet external demands, as opposed to supporting internal objectives (o theuniversity, aculty, department or individual employee). Tis results in expe-

riences o quality work as irrelevant. In other cases, when it is based on indi-vidual ideas and e orts, it is much more likely to be perceived as relevant, but will then o ten take place without or outside a strategic ramework, and willremain well-intentioned, but a more or less isolated project/arrangement.

Te audits also point out that it may be a difcult task to collect the relevantknowledge and documentation and not more than that and to let this beavailable and accessible to the relevant decision makers as well as practitionerso the university when they need it. Tere seems to be a positive developmentunderway in this eld, where registration o in ormation is conducted accor-ding to the needs o the users and external demands at the same time.

All things considered, the universities do not bene t ully rom the resour-ces that are being spent on quality work. And when those involved in it donot see the clear relevance and positive meaning/results o the eforts, thismay easily result in lack o support or the quality work. Te audit panels havestressed that it is very much a job or the management to head the process o developing an institutions quality work in order to signal its importance. Andi it becomes more clear, what the advantages are, and who are responsible orcarrying it out and ollowing up, it is expected to enhance the universitiesactivities and results instead o being regarded as a drain o resources romacademic activities already under pressure.


33/48

FinlandTe Finnish audit model will be under development up to 2007. From 2005to 2007, each auditor and participating higher education institution will be

requested to give eedback on the audit methods and criteria. Te FINHEEC will use the eedback and the discussions with international partners in urtherdeveloping its model or auditing quality assurance systems. As a rule, theaudits are conducted at six-year intervals.

By the end o 2006 FINHEEC had carried out ten quality assurance audits.In December 2006 FINHEEC collected eedback rom both higher educa-tion institutions and auditors concerning the audits. Te ollowing ndingsare based on the responses given by six higher education institutions and 22auditors.

Feedback rom audited higher education institutionsPreliminary in ormation

According to all the responses, the in ormation and instructions provided by FINHEEC were clear. Te negotiations on audit contracts and the prelimi-nary visits to higher education institutions (chair and secretary) were conside-red in ormative. Tey were thought to have especially great bene t in in or-ming the whole sta about the upcoming audit.

Signifcance o the auditing process

All the higher education institutions thought that the auditing process wasuse ul overall and improved their basic operations. Te process clari ed mana-gement and made the whole sta work together. wo institutions stated thatthey had immediately proceeded to initiate development recommended by theauditors. One institution which had undergone an international audit notedthat it had needed and used a lot o translation services and that there hadbeen some problems with the di erent meanings given to certain terms in thediscussions, which were conducted in English

Auditors responses

Most auditors elt that they had received su cient preliminary in ormationabout the audit. On the whole, the content, extent and level o auditor training

were considered good. Te most use ul content had been the concrete matters:audit targets and the criteria to be used; experiences rom pilot auditing; audittechniques; and matters relating to the ormulation o questions to be discus-sed during the site visit. Te auditors appreciated the opportunity to hear theviews o experts who had taken part in previous audits. It was also consideredgood practice to train more than one panel at the same time, because o thechance to share views and experiences.

As regards the preparation o the audit visit, the audit panels saw it vital to

agree on the division o work between them and to study the audit material


34/48

in advance. Te preparation o questions to be posed during the visit was atime-consuming phase. Some o the respondents expressed the wish that FIN-HEEC would provide a selection o questions in support o this work. Meet-ings arranged be ore the visit were considered crucial and it was suggested thatsu cient time be reserved or meetings be ore and during the visit.

Te structure o the site visit days and the choice o the targets to be visi-ted were in general considered appropriate. Some respondents thought thatthe split o the panel into two groups was a good solution, helping to gainan overall picture o the unctioning o the quality assurance system. Somethought the large number o persons interviewed was a problem and suggesteda maximum limit to it.

Some auditors elt that the audit material was airly large and suggested thata maximum number o pages be set. Te audit panels hoped that FINHEEC

would stress to the auditees that the audit material should give a good overallpicture o their quality assurance systems. Te auditors appreciated havingbeen given all the material they had requested to see during audit visits.

Te set o auditing criteria were generally deemed to unction airly well andto make or comparable audits. Te set o criteria also acilitated the writingo the report. However, the auditors saw that the audit targets and audit cri-teria could do with some pruning and removal o certain overlaps. Accordingto some respondents, the challenge is to get di erent audit panels to use thesame scale in applying the criteria.

Te auditors gave positive eedback about the jointly agreed skeleton struc-

ture o the report. Te division o responsibilities in the writing and the jointscrutiny o the report were generally seen to be major strengths. Many respon-dents thought that dra ting the descriptive part o the report be ore the auditvisit was good practice and helped to gain more rom the visit.

As a development proposal concerning the whole auditing process, auditorssuggested that the aims o the audit should be more strongly stressed to thehigher education institutions in advance. Further, the respective roles o theaudit panel and FINHEEC and the decision-making chains should be madeclearer to the auditors. All respondents pointed out the crucial role o theFINHEEC project managers or the success o the whole auditing process. All

the eedback on their demanding work was to be commended. On the otherhand, the auditors noted that it would be in the interest o the whole organi-sation (FINHEEC) i the secretariats employment relation with FINHEEC

were on a more permanent basis and i there were more personnel (secretariat)available or auditing.

Development recommendations to the higher educationinstitutions

In the nal audit report, based on the stated audit criteria and principles, theaudit group appraises the tness or purpose and per ormance o the quality

assurance system, issuing recommendations or its improvement and high-


35/48

lighting best practices. At the end o the report, FINHEEC gives its decisionbased on the audit ndings.

In the light o the audit reports,the strengths of the audited quality assurance systems are long-term work in quality assurance and comprehensive documen-tation o the system and its constituent parts, the whole higher educationinstitutions commitment to developing the quality assurance system, and theinterlinkage o the institutions operational management and quality assu-rance system.

In their development proposals , the auditors have highlighted the role o management in the development o quality assurance and the use o the in or-mation it produces, the extension o quality assurance to all the operationso the higher education institution, and the commitment o the higher edu-cation institution community as a whole, including students, to the quality assurance system.

In its operations, FINHEEC has always emphasised the principle o enhan-cement-led evaluation. Tis means that the evaluations produce in ormationabout higher education and its quality which can be used in institutionaldevelopment. Tis in ormation is also used by the Ministry o Education, orexample, in per ormance-steering and decision-making.

IcelandExternal evaluations during the last 12 years have shown that the higher edu-

cation institutions in Iceland o ten seem not to have been able to change theirinternal quality work in accordance with the enhancement-led evaluations theinstitutions had been a part o . Te ongoing change o laws and regulations,resulting in the 2006 Higher Education Act can partly be seen as a reactionto that result.

As Iceland has just started her cycles o accreditation with quality audits asa part o the ollow-up process, it is too soon to draw any great conclusions o the process and such an act could in many ways be misleading. However, itis worth mentioning that the ongoing accreditation procedure seems to haveindicated a new and better notion o the importance o internal quality work

within the higher education institutions.

Norway35 institutions had had their quality assurance systems audited by NOKUby the end o 2006. 31 o them were approved. O the remaining our, threehad been re-audited with a positive outcome. Te audits generally indicatethat implementing the quality assurance systems has required a longer timeperiod than expected. At the end o 2006, many institutions had not imple-mented them ully. Hence, the audit panels and the Board o NOKU are


36/48

o ten aced with the problem o nding a balance o what is a su cient degreeo implementation.

Recommended areas or urther development in the auditreports

Te requirement to prepare annual institutional reports on quality work ismet, on the whole. O ten they are based on quality reports rom several levels

within the institution. However, they are o ten also criticised or not provi-ding analytical accounts o educational quality and proposals or action to betaken. Tis is partly due to the act that systems have only recently been, orare not yet ully, implemented. Hence, the available data has been insu cientor the time-span has been too short.

Quality assurance systems are o ten complex. Describing them in a com-prehensible manner, which makes it possible also or those who are not centralparticipants in quality work is not an easy task. And yet it is important thatstudents and external interest groups should be able to have an understandingo the institutions quality work.

Te audit panels requently have remarks on the institutions way o ensur-ing broad participation in Quality work. Sometimes the student representa-tives or the teachers are not involved in analysing the in ormation gathered

rom the quality assurance systems to a su cient degree.Te quality assurance systems o ten do not include student activities in

practice training periods to a su cient extent. Quite o ten the systems seem

to cover ordinary study programmes better than the various short time coursesthat are nanced outside the basic state grant. Also, the eedback rom parttime students attending web-based provision and eedback rom Ph.D stu-dents is o ten insu ciently covered in the quality assurance systems.

Students evaluation o teaching and learning is an important part o allthe QA- systems. Te panels o ten have remarks on the questionnaires in use.Student participation in evaluation during a term seems to be well taken careo but many institutions lack su cient arrangements or systematic eedback to the students a terwards.

Follow-up o quality work is mentioned as a development area. In many

cases planned improvement activities have not been implemented. Reportsindicate, however, that several institutions have positive experiences o peerguidance, educational development and various routines or ollow-up o acti-vities that may be help ul in various situations.

Feedback rom institutions

In a project on expert knowledge, NOKU has analysed the ormal state-ments institutions have given a ter the audit. Out o the 35 statements, only 3raise a discussion on the principles o the audit and its methodological validity.

wo institutions analyse how educational quality is made an integral part o

the institutions strategic work (one o the evaluation criteria). Te audit panels


37/48

may transgress their mandate by evaluating the strategy o the institution inthe light o educational policy. In Norway, this is the duty o the institutionitsel , and the Ministry discusses such topics in its steering dialogues with thestate-owned institutions. Also, with regard to this aspect, there is a ne linebetween aspects o an audit and aspects o institutional accreditation.

Sweden As indicated in the description o the Swedish audit models, the criteria on which the judgements in the reports o the rst two rounds o audits in Swe-den were base on the ollowing aspects: Strategy or implementation o quality assurance processes Academic leadership

Participation by all the sta in quality enhancement and assurance Integration o quality work in all activities at the institutions Evaluation and ollow-up activities Internationalisation External pro essional relations Gender equality

Te audits later came to identi y a structure o quality work based on theattainment o our levels, each o them presupposing the previous one: Establishing aims o Quality work

Planning and implementing activities to meet the aims Evaluating results o implementation aking measures to improve and integrate Quality work on the basis o

evaluations.

In a study o the rst round o audits based on a reading o 36 reports, Sten-saker (1999) uses the various recommendations as a means o identi ying per-ceived weaknesses in institutional quality assurance with regard to the aboveaspects. It turned out that most o the recommendations concerned leadershipand organisation o Quality work (21 per cent) and then in descending order;

policy and strategy (14 per cent), universal participation in Quality work (12per cent), evaluation and ollow-up (12 per cent), staf development (12 percent), cooperation with stakeholders, internationalisation (4 per cent) and gen-der equality (3 per cent).

Te impact o audits (or rather o the internal Quality work o the institu-tions) may be seen in a similar investigation our years later. Te interpretationassumes that a recommendation is seen as a negative view o the aspect andthat positive impact can be measured in terms o a relative decrease o recom-

. Summary o part o Wahln .Summary o part o Wahln .

. It should be noted that some o these did not appear in the original list o aspects but wereadded by the auditors.


38/48

mendations. It showed that the percentage o recommendation with regardto leadership had gone down to 14. A similar development could be seen withregard to the percentage o stakeholders, which was down rom nine to six percent. On the other hand, a negative development was seen in participation inQuality work (up rom 12 to 19 per cent).

A closer reading o reports indicates that auditors had seen positive develop-ments in a number o institutions, Tus they saw improvements in leadershipand organisation in 24 o the 36 institutions, in student infuence in 15 insti-tutions, policy and strategy in 14 institutions, cooperation with stakeholdersin 13, and evaluation and ollow-up in 13, educational development in 12 andinternationalisation in 12 institutions.

With regard to living up to the quality cycle (planning implementing ollow-up evaluating new planning) the investigation concludes thata ter two rounds o audits, only about one-third o institutions had developedsystematic processes or ollowing up and evaluating activities. It thus takesconsiderable time to achieve a complete change in this direction through qua-lity audits.


39/48

Strengths, challenges and developments

General observationsTe development o systematic quality work in higher education institutionstakes time. Tis may be deduced speci cally rom the experiences in Den-mark, Norway and Sweden. Functioning quality work is not always at handin all institutions even a ter many years. And it still does not have the ullacceptance o the academic community. One reason or this may be that thesystems that have been developed have not been capable o making li e easier

or the teaching sta . Instead, quality work is still seen as a burden on top o everything else required o a university teacher, at a time o dwindling resour-

ces and lack o time or research.Secondly, and more importantly, the link between systematic quality work

and improved quality o provision and outcome, output and impact is unclear.Tere is little, i any, incontrovertible evidence that it leads to higher quality.

We do know that it involves routines that may contribute to, among otherthings, better management, leadership and decision-making, better know-ledge o the opinions and per ormance o students and sta , better relations

with internal and external, including international, stakeholders and partnersand, last but not least knowledge o the quality o provision. But the link isstill not obvious and thus the validity o the model may be challenged.

Yet, the experience o the Nordic countries o demands or quality work and or national audit o the institutions eforts is, on the whole, positive.In an international context, it helps institutions to live up to the EuropeanStandards and Guidelines, which will acilitate international exchange andcollaboration.

It is reported that site visits in audits inspire discussions on the quality o teaching, research and management and quality work within the whole insti-tution to be evaluated. Te audit panels report will always include recommen-dations on urther development o an institutions e orts to improve educa-tional quality. Audits also contribute to transparency o institutions quality operations through demands or systematic documentation and their insis-tence that the sta should work in the direction o joint quality goals. Mostimportantly, perhaps, these demands also help institutional leadership at vari-ous levels to set priorities and thus to make e cient use o the institutionsresources. Hence, the model holds the potential o inspiring the quality cultureo the institutions o higher education, and is appreciated by many institutions

or this reason.One area that has been addressed is the costs o quality work and, in par-

ticular, quality audit. Tis is a discussion that o ten arises in the context o external quality assurance. One answer is that it is important or students,prospective students, stakeholders and those who und higher education to


40/48

know the quality o what is provided and whether money is well spent, notleast in international comparison. Tus, external assurance is important, butit ought to be at a reasonable level and costs in terms o money and e ortsmust be reasonable in view o the total costs o higher education. Audit callsattention to and puts the responsibility or quality where it belongs, i.e. at thehigher education institutions, and allows them to demonstrate that they havethe tools to assure and enhance their provision. For both quality assuranceagencies and institutions it is probably the least costly orm o external inter-

erence in comparison with subject and programme evaluation.

Observations on elements o the methodologyThe sel -evaluation

As has been described in the section on sel -evaluation, not all the Nordiccountries require sel -evaluation. It may be argued that there is no substantialdi erence between requiring a sel -evaluation process and report and requi-ring existing material. Te material demanded by the agencies in Norway and Finland is such documents as describe quality assurance procedures anddemonstrate their impact. It is required that on the basis o a quality pro-gramme, activities should be in place (evaluations, taking account o per or-mance indicators, procedures based on interpretations o in ormation derived

rom the results o quality work, etc.). Tis is meant to make such procedu-res and processes a normal part o everyday academic li e, and to put less o a

burden on institutions.On the other hand, the sel -evaluation process required in Denmark, Ice-

land and Sweden is intended to activate the institution as a whole and at bestcontribute to the improvement o processes and results. It is a process requiredonce every ve or six years depending on the length o the evaluation cycle.It is a major e ort by the whole institution and may thus have a lasting e ecton quality work. I the institution has a well unctioning quality assurancesystem, it is a matter o putting what already exists together in a report, whichis part o a regular process. In those Nordic countries where sel -evaluationsare not mandatory an annual report on quality work is demanded, which inmany ways resembles what is required in a sel -evaluation. It may, there ore,be argued that the di erences between the two models are not overwhelming,provided that systems are in place.

The criteria

Te use o prede ned criteria or decisions based on external evaluation isrequired by the European Standards and Guidelines. Tey are a condition orinstitutions to know on what grounds they will be assessed, and or expertpanels to know on what they are going to base their judgements. At the sametime, a narrow interpretation o criteria by institutions and expert panels may lead to unwanted standardisation o quality work. Hence, it is pre erable that


41/48

criteria should rather be gen

Documents

Quality Audit in the Nordic Countries