FOR THE FURTHER - European Commission...Higher Education Area (ESG) and on the intercultural awareness of internal quality assurance (IQA) with the development of common methodologies

EIQAS ENHANCING INTERNAL QUALITY

ASSURANCE SYSTEMS ERASMUS+ PROJECT

2014 – 2016

REPORTONDRAFTFRAMEWORK

FORTHEFURTHER

INTERCULTURALTRAININGOFEXPERTS

ANDARRANGEMENTSFOREXCHANGEOFEXPERTSOF

QUALITYASSURANCE(QA)AGENCIES

2

This publication has been funded by the European Commission as part of the Erasmus+ programme. The European Commission’s support for the production of this publication does not constitute an endorsement of the contents, which reflect the views of the authors only, and neither the Commission nor the National Agency of the Erasmus+ programme can be held responsible for any use, which may be made of the information contained herein. FREE PUBLICATION

3

Table of Content

Part 1 Framework for the further intercultural training of experts…………………………………………………….5

1. Introduction…………………………………………………………………………………………………………………………5

2. Starting Point………………………………………………………………………………………………………………………..6

2.1. ECA tools………………………………………………………………………………………………………………………7

2.2. ENQA tools…………………………………………………………………………………………………………………..7

2.3. EUA activities…………………………………………………………………………………………………………………7

3. Methodology…………………………………………………………………………………………………………………………7

4. The experiment of intercultural training in the EIQAS project……………………………………………………8

5. The results: standard by standard………………………………………………………………………………………….…9

ESG 1.1. Policy for quality assurance………………………………………………………………………………………9

ESG 1.2. Design and approval of programmes………………………………………………………………………10

ESG 1.3. Student-centred learning, teaching and assessment……………………………………………………11

ESG 1.4. Student admission, progression, recognition and certification…………………………………….12

ESG 1.5. Teaching staff………………………………………………………………………………………………….……13

ESG 1.6. Learning resources and student support……………………………………………………….…………13

ESG 1.7. Information management………………………………………………………………………………………14

ESG 1.8. Public information……………………………………………………………………………………………..…15

ESG 1.9. On-going monitoring and periodic reviews of programmes……………………………………..…15

ESG 1.10. Cyclical external quality assurance…………………………………………………………………………16

6. A critical analysis of the results of the workshops……………………………………………………………………16

Part 2 Working arrangements which QA agencies may adopt for future exchange of experts………….…18

Annexes…………………………………………………………………………………………………………………………………...24

Annex 1 Training Event 1 Warsaw, Poland, 26th – 30th June 2015……………………………………………………25

General description SEMINAR ON IQA & ESG PART 1……………………………………………………………25

Programme…………………………………………………………………………………………………………………………….…26

Parallel workshops scenario (27th of June)……………………………………………………………………………………..28

Annex 2 Training Event 2 Ljubljana, Slovenia, 7th – 11th December 2015…………………………………………31

Programme STUDENT TRAINING ON IQA & ESG PART 1……………………………………………..….…31

Annex 3 Training Event 3 Lisboa, Portugal (A3ES), 18th – 22nd January 2016……………………………………34

Programme………………………………………………………………………………………………………………………….……35

Parallel workshops scenario (18th of January)…………………………………………………………………………………38

Workshop 1. (18th of January) Preparing the site - visits of day 2. Questions……………………………………..38

Outcome of Workshop 1: questions to the HEIs………………………………………………………………………….41

4

Workshop 2. (20th of January) Have you got the answers?

Positive and negative aspects of each HEI visited. How are IQAS delivering?.........................................46

Outcome of Workshop 2: Have you got the answers?

Positive and negative aspects of each HEI visited………………………………………………………………………..…49

Workshop 3. (20th of January) IQAS and the Standards…………………………………………………………………60

Workshop 4. (20th of January) Playing our role 1st round……………………………………………………………..…63

Workshop 5. (20th of January) Playing our role 2nd round……………………………………………………………..…64

Workshop 6. (21st of January) The magic stick workshop:

The link between IQAS and the QA of teaching and learning. How are IQAS delivering?....................65

Authors: Madalena Fonseca, A3ES / Izabela Kwiatkowska-Sujka, PKA

Editorial Board: Madalena Fonseca, A3ES; Izabela Kwiatkowska-Sujka, PKA; Klemen Šubic, Jernej

Širok, SQAA; Mila Penelova, Todor Shopov, Hristina Savova, NEAA1

1 List of used abbreviations: A3ES Agência de Avaliação e Acreditação do Ensino Superior, Portugal PKA Polish Accreditation Committee, Poland NEAA National Evaluation and Accreditation Agency, Bulgaria SQAA Slovenian Quality Assurance Agency for Higher Education, Slovenia

5

Part 1 Framework for the further intercultural training

of experts

1. Introduction

QualityAssuranceAgencies(QAAs)inEuropearecollaboratingcloserandcloser,throughvarioustypes

ofpartnerships,asisthecasewithformalassociationssuchasEUA,ENQA,EURASHE,EQAR,CEENQA

or ECA and other arrangements for special projects with the aim of developing common Quality

Assurance(QA)toolsandmethodologies.

InternationalisationisthereforecurrentlyamajorstrandintheaccreditationagenciesinEurope.The

Erasmus+project ‘Enhancing InternalQualityAssuranceSystems’–EIQAS - isoneexampleof such

initiativesandranfrom2014totheendof2016.EIQASisajointinitiativeofQAAs,rectors’conferences

andHEIsoffourcountries:Poland(co-ordinator),Portugal,BulgariaandSlovenia.

TheoriginalityofEIQASapproachanditspotentialaddedvaluetoQAinEuropeliesinthequestfor

a common understanding of the Standards and Guidelines for Quality Assurance in the European

HigherEducationArea (ESG)andonthe interculturalawarenessof internalqualityassurance (IQA)

withthedevelopmentofcommonmethodologiesforexternalqualityassurance(EQA)andtransparent

informationprovisionbythedifferentstakeholders(HEIs,students,QAAs,experts,etc.).

One of the core activities of the EIQAS project were the training events that took place in three

differentcountriesfromthepartnersoftheconsortiumandweretargetedtothreedifferenttypesof

stakeholders;namelyhighereducationinstitutions,studentsandexternalexperts.

Thisreportingguidehasbeendevelopedwiththeaimofdesigningaframeworkfortheintercultural

trainingofexternalexpertsforexternalqualityassuranceprocessesandincludesthemethodologyand

theresultsoftheinterculturaltrainingactivitiesoftheproject.

Theguide for the intercultural trainingofexperts is alsoexpected todisseminate some innovative

outputsoftheEIQASprojectthatcanbeappliedelsewhere.

During the training events of the EIQAS project intensive discussions on the partner Agencies’

methodologies for the assessment of IQA took place and, where possible, participants arrived at

a common understanding of Part 1 of the ESG and identified key issues and features of best and

innovativepracticeinIQA.

TheEIQASprojecthadtwomainobjectives.OneofthemwastoincreasethecapacityofHEIsinthe

participatingcountriestodevelopIQAandbuildaqualityculturebyenhancingtheirawarenessand

understanding of Part 1 of the ESG and thereby identifying, developing and disseminating good

innovativepracticeininternalqualityassurance.

ThesecondobjectiveofEIQASwastoincreasethecapacityoftheparticipatingQAagenciesinEQAby

comparingtheirmethodologiesfortheassessmentofinternalqualityassuranceandexchangingand

developinggoodpracticeinIQAassessment.AreferenceframeworkforacomparativeanalysisofIQA

assessmentmethodologieswasdevelopedatthebeginningoftheproject.Later,thepartnershadthe

opportunity for face-to-face discussions and in-depth comparative analysis of the Agencies’

methodologies. Conclusions provided the basis for a comparative report on IQA assessment

methodologiespostedonthewebsiteoftheproject.Agencieswhichhadnotworkedcloselytogether

prior to this gained a goodopportunity to get to knoweachotherbetterwith a view to initiating

exchangeofexpertsandconcludingbilateralcooperationorrecognitionagreementsinthefuture.

6

2. Starting Point

TheStandardsandGuidelinesforQualityAssuranceintheEuropeanHigherEducationArea(ESG)are

consideredthemostimportantmilestoneintherecentdevelopmentofQAsystems.Launchedin2005,

theycoverinternalandexternalQAandQAintheaccreditationagencies(QAAs).Tenyearsafterthe

launchoftheESG,inJuly2015,arevisedversionwasapprovedatYerevan,Armenia.TheESGare,as

thenameindicates,guidelinesorreferences,subjecttocustomapplications.

Quality, however, is still not an easy concept when it comes to QA in HE; there are different

understandings of the concept of quality, not only by the different stakeholders, but throughout

individualhighereducationinstitutions,countriesorevenregions.Theperceptionof“quality”varies

amongstakeholdersandacrossscientificareas.Stakeholdersarealsoverydiverse;suchasinthecase

ofstudentsandsotheperceptionof“quality”variesalsoinsidegroupsofstakeholders.

Within Europe, the tendency and the actual practice in the accreditation processes is that the

evaluation panels should include an expert fromabroad. Therefore, intercultural trainingmakes it

easierfortheexchangeofforeignexpertsintheevaluationpanels.Infact,oneoftheessentialparts

ofeachQualityAssuranceAgency isagroupofwell-trained independentexpertswhoperformthe

externalevaluationsat theHEIs.Thoseexperts includeacademics,professionalsandstudents.The

mainadvantageof the intercultural trainingevents is theexchangeofgoodpracticesbetween the

expertsfromdifferentEuropeanQAAs,therepresentativesoftheHEI’s,andstudents.

ThemainoutputoftheEIQASprojectistheGUIDEtoIQApublishedonthewebsiteoftheprojectas

anebook.TheGUIDEtoIQAshouldbeconsideredaroadmapforfurthertrainingeventsandisthe

pointofdepartureforthepresentframeworkfortheinterculturaltrainingofexperts.

Infact,oneofthemostpraisedresultsoftheEIQASprojectwasthedevelopmentofanintercultural

approachonIQAandEQAand,inthiscontext,theframeworkisacomplementtotheGUIDEtoIQA.

Thisframeworkincludesexamplesandsuggestionsfortheorganisationoftheinterculturalcomponent

ofthetrainingofexperts forQAAs.TheAnnexesofthis framework includenotonlytheformsand

programmesofthetrainingevents,butalsosomeoutputsoftheworkinggroupsoftheintercultural

workshops organised for the discussions. This material should be considered quasi-confidential

becausemost of it corresponds to texts thatwere elaborated in a short space of time during the

workshops,ratherthanworkedoutafterwards.Theyshouldbetakenasexamplesoftheoutputsthat

canbeexpectedfromsuchaprogramme.Participationofallstakeholdersandtraineeswascriticaland

theoutputsareagoodmirroroftheverydynamicdiscussionsthattookplace.Mostoftheworkshops

were quite turbulent and the participants had some difficulty fitting in time for coffee breaks. By

publishingthismaterial,weaimtoinspireothersandleadbyexample.

Itisworthwhiletonotethatthisframeworkisalsoacomplementtootherexperts’trainingsschemes

anddoesnotcorrespondtoacompletetrainingpackageinitsownright.

AllQAAsinEuropeorganisetheirowntrainingandbriefingofexperts.AssociationsofQAAsandsimilar

institutions,(as isthecaseofECA,ENQAorEUA,amongothers)alsoorganisetrainingfordifferent

stakeholdersandaccreditationprocesses.Thepresenttool(theframeworkforinterculturaltraining)

shouldbeusedtodevelopspecificmodulestobeincorporatedintheothertrainingprogrammes.

There are several types of training for the different stakeholders and for the different types of

assessments: ex-ante accreditation of programmes; regular accreditation and re-accreditation of

programmes; institutional (or system) accreditation; and audits or certification of internal quality

assurancesystems,amongothers.Eachtrainingprogrammehastobedesignedtakingintoaccount

thetargetgroup,thetypeofassessment,thenationallegalframeworksandtheguidelinesinusein

thedifferentQAAs.

Globaltrainingprogrammesandschemescanbefoundinthewebsitesofthoseinstitutions.Beloware

someexamples.

7

2.1. ECAtools

TheEuropeanConsortiumforAccreditation–ECA–developedaPortalfortheorganisationoftraining

forexperts,thatincludesseveraltoolsintheformofpublications:

§ ECAGuidetoGoodPracticesforTrainingofExperts

§ ECAGuidelinesforTrainingofExperts

§ Recommendationsfortrainingofexperts

§ ECAHandbookfortheTrainingofPanelMembers

Thepublication“ECAGuidelinesforTrainingofExperts”istheoutcomeoftheECA'sE-TRAINproject

andpresentsguidelinesandadviceonthewayforwardinthedevelopmentoftheECA'strainingof

expertsthattakepartinexternalqualityassuranceprocedures.

Http://ecahe.eu/w/images/c/ce/ECA_Publication_-_Guidelines_for_Training_of_Experts.pdf

2.2. ENQAtools

The European Association for Quality Assurance in Higher Education (ENQA), in association with

EQUAR,offersregulartrainingofexpertsforvarioustypesofassessmentsandaccreditationaswellas

differentstakeholders:

http://www.enqa.eu/index.php/reviews/training-of-experts/

2.3. EUAactivities

TheEuropeanUniversityAssociation (EUA)alsooffers several tools, research resultsandorganises

activitiestosupportexperts’training.

http://www.eua.be/activities-services

3. Methodology

The EIQAS project included 3 training events; one for representatives of the HEIs, the second for

studentexpertsinvolvedinQAandthethirdoneforexperts(academicsandprofessionals).Therewere

approximately40participantsperevent;oneortworepresentativesoftheinvolvedstakeholdersper

countryandprojectpartnerandotherlocalguestsfromthesamestakeholderstype.

Eacheventtook5daysandincludedseveralworkshopswithinteractiveparallelsessionsinworking

groups(WG)encouragingpeopletoengageindiscussionandsharetheirviewsonPart1ESG.

Eachworkshopgathered together representatives from the four countries inorder to facilitate an

interculturalexperimentduringeachsession.Participantsmovedfromoneworkshoptoanother in

severalroundsofsessionsinsuchafashionthattherewererepresentativesofthefourcountriesinall

thesessionsandacrossalltheworkinggroups.

Annex1,2and3correspondto theProgrammesandGuidelines for theParallelworkshops,of the

threetrainingeventsofEIQAS.

Each roundof eachworkshophada rapporteur, selectedamong theparticipantsbelonging to the

partnersof theproject.Themaingoalsof theworkshopswere toenrich thequantitative research

whichwasconductedasanonlinesurveyonPart1oftheESGandtogatherstakeholders’perceptions

throughthecontributionsoftheparticipants.

8

Goingthroughthestandards,onebyone,theperceptionsgatheredontheinterculturalworkshopsof

thetrainingeventsoftheprojectwereanalysedinacriticalwayandconfrontedwiththetoolsand

methodologiesoftheQAAsofthedifferentcountriesinordertoenrichtheGuidetoIQAasthemain

outputoftheproject.

Theproject partners developed a set of focus questions that structured the different intercultural

workshopsofthetrainingeventsforthedifferentstakeholders:

1. Whatisthemostimportant/crucialpartofagivenstandard/orguidelineshavingapotential

impactonthequalityenhancement/oronthedevelopmentofqualityculture?

2. What are themain obstacles in the implementation of given standard andwhy? Are they

derivedfromthecontentofthestandard?

3. Whataretheneedsforfurtherinterpretationofgivenstandards?Whichaspectofstandards

areunclearforHEIandwhy?

4. Doyouidentifyanygoodpracticeinrelationtoagivenstandard?

In a broader sense the focus questions should have been considered as a general framework for

experts’reflectionsstimulatedduringtheworkshops.Thequestionswerealsointendedtoenhance

discussionamongparticipantsofworkshopsaswellasallowingthemtoexchangetheirexperiences

and views.Workshops’ rapporteurs/moderatorswere responsible for facilitating the discussion on

eachstandardofESGbygivingahelpfulhandwheneveritwasneeded.Additionally,therapporteurs

ofeachworkshophada“Moderatorschecksheet”intheguiseofaguidelineformtodocumentthe

resultsofthediscussion,delivertothecoordinatorsandpresentonageneralmeetingoftheevent

(Annex 1 and 3). It should be noted that the workshops concept relied on a brainstorming and

benchmarking approach to training thatwas rather softly guided bymoderators. The intercultural

aspectswereemphasisedintheprocessoftrainingtobuildawarenessofsimilaritiesanddifferences

inunderstandingthesamestandardsofESG2015.

TwotrainingeventsoftheEIQASprojectincludedasitevisittoseveralHEIsandtheparticipationina

role-playingexperience(Annex2and3).Thiswasrootedinlearningbydoingmethodsofteachingthat

were intended to stimulate theactiveparticipationof trainees inall activities. Practicewasa core

approachtothetrainingofexperts.Furthermore, itrequiredteamworkactivity intheinternational

andinterculturalprovisions.Theresultsofthosetrainingeventswerethereforemorecomprehensive

inwhatconcernstheperceptionsofthestakeholders.

4. The experiment of intercultural training in the EIQAS project

Overall,consideringthecountryreportsbasedontheinitialcomprehensivequestionnaireoftheEIQAS

projectandtheresultsofthe3trainingeventsandtheperceptionsoftheparticipants,itshouldbe

emphasisedthatthereexistsagoodstandardofknowledgeoftheESGandallstakeholdersexpressed

thecapacityforanadequateinterpretationofESGstandards.

The main challenge to the widespread application of ESG is on account of some barriers for the

effectiveinvolvementofsomestakeholders,especiallyteachersandstudents.Themostmentioned

priorityactionswerethetrainingofexpertsforthepanelsandthegeneralisationofthepreparation

andpublicationofreportsinEnglish.

Theaforementionedtwotopicsevidencetheimportanceoftheinterculturalcommunicationamong

qualityassuranceagenciesanddifferentcountries.Thisisnotonlyforthepurposeofaccreditationof

jointprogrammes,butalsotosupportalargesetofrecognitiontoolsandprocedurescrucialforthe

mobilityofstudentsandgraduatesandfortherecognitionof theirqualifications,programmesand

provisions.

9

Whatfollowsaretheresultsofthetrainingeventsandthecontributionsoftheparticipantsworked

outintheframeworkoftheESG.

The remainder of this framework is included in chapter 5. The results of the experiment of the

interculturaltrainingofexpertsisorganisedstandardbystandard(PartIofESG).Foreachstandarda

shortsnapshotofthestateoftheartinthefourcountriesoftheproject,basedonthecountryreports

andonthe initialcross-countryreport, ispresented.Theresultsof the interculturalworkshopsare

summarisedthereafterandincludethecontributionsaddressingthefocusquestionsworkedoutat

theworkshopsandalistofpotentialquestionsonthecorrespondingstandardforasitevisit.

Therelevanceofthesematerials lies inthefactthatthesearetheresultofan interculturaldebate

carriedoutinseveralroundsofdiscussionbygroupsofparticipantsfromthe4countriesoftheproject.

Thedebateswerecomplexduetothenecessitytoaccommodateadiverserangeofperspectivesand

perceptionsinordertostrivetowardsacommonconclusion.

Beyondthe10ESG,threeotherstandardscreatedbythePortugueseAgencyinitsauditmodeltoIQAS

havebeenfurtheranalysed,althoughinamoresuperficialway;thosethreeextrastandardscoverthe

followingareas:research,internationalisationandrelationswithcommunity.

Thisreport/frameworkincludesonlytheresultsofthetrainingeventsinordertotestthemethodology

andofferasnapshotofit.Theauthorsdidnotaddanynewquestionsortopicsregardingrelevance,

barriers or good practices beyond those gathered in the training events. Some arrangements and

mergesofsimilarsentenceswereconducted,withdifferentwordingsbutnofreshcommentswere

added.AbroaderdiscussionanddeeperanalysiswascarriedoutfortheGUIDEtoIQA.

Theaimofthepresentreportorframeworkistohighlightthetopicsorissueswhichreceivedthemost

attentionfromthedifferentstakeholders,representedbytheparticipantsonthetrainingeventsof

theprojectandcanbecomplementedwiththeotheroutputspostedonthewebsiteoftheproject.

Itisintendedthattheframeworkshouldbeusedasaguidelineoramodelforfurthertrainingevents.

It ispresentlyanongoingprocessandnotyetareadytooltouseorastandardtoadopt.Itaimsat

beinganexamplewithresultsthatarealreadybeingusedinsomeoftheparticipantcountries.

5. The results: standard by standard

ESG1.1.Policyforqualityassurance

Approximately60%oftheHEIsthatrespondedtothesurveythattookplaceinthefourcountrieshave

a published policy for quality assurance and external stakeholders were involved in both the

development and implementation of the policy. In most cases quality strategic goals are directly

incorporatedbyHEIsintostrategicplansandtheydonotdevelopseparatedocumentsfortheirquality

assurancepolicy.Cooperationwithexternalstakeholdersshowsasanincreasingtrendforthelastfew

yearsandconfirmstheresponseoftheHEsectortolabourmarketneeds.Thisstandardcorresponds

totheprocedureforinitiating,discussing,implementingandassessingQAstrategyand/orpolicy.The

institution’s policy for QA must be visible and planned for all dimensions of the HEI's operation

(teaching, learning, research, collaboration with environment, strategic planning, involvement of

internalandexternalstakeholdersetc.).

Inwhatconcernsrelevance,obstacles,furtherdevelopmentandgoodpractices(topicsaddressedin

the focus questions), all stakeholders recognise that Quality Assurance should be part of the

institutional strategy; a policy of continuous improvementmust be embedded in theHEIs.Not all

institutionshaveorneedacomplexandformalsystem. Instead, internalqualityassurancesystems

shouldbetailoredandrespondtotheneedsandexpectationsofthestakeholders;themostimportant

issue is the development of a Quality Culture visible across all levels and incorporated into the

academictradition.Theinvolvementofexternalstakeholdersisveryimportantinthisregard.

10

Some risks have been identified. For example, HEIs may establish very complex and not flexible

structuresforQualityManagementanddominantconservativeculturemayblockthedevelopmentof

moreinnovativestructures.

Guidingquestionslistedforasitevisit,facingstandard1.1:

↘ WhatisyourframeworkforbuildingyourIQAS?

↘ HowisyourQApolicylinkedtothestrategicmanagementofyourinstitution?

↘ Whocreated themission, thevisionand theQAmission?Didall the stakeholdersparticipate?

Students?

↘ Arethestrategicgoalsinlinewiththemission?

↘ Howareinternalandexternalstakeholdersinvolvedinthedevelopmentandimplementationof

yourIQAS?Whatarethegoodpracticesinthisarea?Doyouhaveanyinnovative

methodologies?

↘ HowistheeffectivenessoftheIQASassuredandmeasured?

↘ Howdoyouguaranteethatallinternalandexternalstakeholdershaveacommonviewand

understandingofyourqualityassurancesystem?

ESG1.2.Designandapprovalofprogrammes

Theprocedurefordesignandapprovalofprogrammesisnotfollowedinallfieldsofstudyinallthe

HEIsofthefourcountriesoftheproject.Whiletheapprovalproceduredoesnotgenerateaproblem

for respondents, the designing procedure is considered to be a weakness. Nevertheless, most

programmesaredesignedinlinewithinstitutionalstrategies,missionstatementsandvisionandrefer

totheNationalQualificationFramework.Theinvolvementofexternalstakeholdersintheprocedure

of design and approval still requires more attention. External stakeholders include, for example,

alumni,graduates,employersfromthebusinessorpublicsector,otherbusinesspartners,mentors,

etc.

Inwhatconcernsthisstandardthereareimportantbarrierstoimplementation.Therearedifficulties

inthedefinitionandvalidationoftheintendedlearningoutcomes;difficultieswiththeinvolvementof

externalstakeholders(suchasgeographicaldistancesbetweenHEIsandexternalstakeholders);and

variabledesignofprogrammeswithoutinternalconsistencybetweenthecoursescanbefoundmore

oftenthandesired.

ThemismatchbetweenHEIsandtheexternalstakeholderstakesdifferentaspectsandexistsatseveral

levels: both stakeholders have different aims, different timelines and, above all, speak different

languages.Accordingly someaspectsof the standardneed further clarification inorder to fostera

commonunderstandingoftheconceptsbyallstakeholders.

Procedures,institutionalregulationsandothermechanismshavetocomplywiththenationallaw;in

somecountriesthereisadegreeofover-regulation,frequentchangesofthenationallegislation,too

muchbureaucracyandlackofautonomyofHEIs.Thoseaspectsarealsoconsideredimportantbarriers

tothedevelopmentofIQAS.

In termsof relevance, further development and goodpractices, stakeholdersmentioned thatHEIs

needtwoproperlyconstructedandimplementedprocessesfordesignandapprovalofprogrammes.

Firstly,thedesignoftheprogrammesmusttakeintoaccountthedefinitionofthelearningoutcomes

in line with the qualification framework (national and European), as definition and validation of

intendedlearningoutcomesrepresentacrucialpartofthestandard.HEIsshouldbecomeawareofthe

fact thatestablishing intended learningoutcomes represents 'apromise' to thestudentsandHEIs

11

must thus ensure that the intended learning outcomes have been achieved. Intended learning

outcomesshouldencompassallthreeaspects–theknowledge,skillsandcompetencesobtained,and

notmerelyfocusonthecompetencesachieved.Recognitionofformalandinformallearningshouldbe

improved in linewith the implementationof theEuropeanandNationalQualificationFrameworks.

(Althoughthisissueispartlyrelatedtothestandard1.4).

Secondly,internalandexternalstakeholdersshouldbeincludedintheprocess.Internalstakeholders

should be represented by not only teaching staff and students but also administrative staff.With

regard to external employers, special attention should be devoted to alumni, most of whom are

familiarwithboththeuniversity(anditsprogrammes)and,ontheotherhand,thecurrentsituationin

thelabourmarket.Theissueofemployabilityshouldbetakenintoaccountwhenadoptingthiskindof

strategicapproach.Teachingmethodsarealsorelevanttotheachievementoflearningobjectivesand

outcomes.

Theaspectofinternationalisationshouldbeconsideredaspartofthestandard(amissingtopic).

Acyclicallinkshouldbeestablishedbetweenstandards1.2and1.9.


↘ Whoisinvolvedindesigning,approvingandimplementingyourQAprocedures?Whatisthe

procedurefordesigningandapproval?

↘ Howaretheexternalstakeholdersinvolvedinthedesignoftheprogramme?Anyexamples?

↘ HowaretheLOsconceptsimplemented?HowdoyoucheckthealignmentofILOswithNQFand

EQF?

↘ Howdoestheinstitutionrelatethelearningoutcomeswiththenationalqualification

framework?HowdoyouverifytheachievementofILOs?

↘ Whyandwhendoyoudecidetolaunchanewprogramme?

↘ Howaretheprogrammeslinkedtolabourmarketneeds,socialneeds,andthedevelopmentof

thescientificdisciplines?

↘ Whatisthesystemfortheidentificationofachievedlearningoutcomes(onlyteachers,external

examiners….)?

↘ Howdoyouvalidatethelearningoutcomes?(BothintendedandachievedLearningOutcomes).

ESG1.3.Student-centredlearning,teachingandassessment

Students are systematically involved in IQA activity, and student-centred learning, teaching and

assessment issaidtobewellunderstoodbyHEIsallacrossthefourcountries.However,theactual

implementationofthisrangeofissuesintherevisedESGshouldbemorefullyaddressed,accordingly

totheresultsofthecountryreports.

Fairness and consistency achieved through the procedure of student assessment was quite

misunderstoodbymostHEIsandover62%respondentsdecidedtoskipthequestion.

In what concerns relevance, obstacles, further development and good practices, all stakeholders

recognisethattheparticipation,collaborationandfeedbackofstudentsiscriticaltothedevelopment

ofstudent-centredlearning.Institutionalsupportforgoodteachingpracticesandregularpedagogical

monitoringarerelevant.Theinstitutionshouldstimulatethemotivationoftheteacherstoimprove

theirpedagogicalmethodsanddevelopflexible learningpaths.The institutionshouldstimulatethe

dialogue and self-reflection of students and their engagement in research activities. Some have

instigated several good practices, including the creation of commissions for teaching and learning

pedagogy,andprojectsinvolvinginnovationlaboratoriesforteachingandlearning,amongothers.

12


↘ Howdoyouencouragestudentstowardsactivelearningandaself-reflectionprocess?

↘ Howdoyoumotivatetheteacherstowardschangingtheirapproaches,methods,innovationin

teaching,i.e.,theirteachingstrategy?

↘ Howdidyouchangetheassessmentproceduresinordertomeetthestudents’activerolesin

creatingthelearningprocess?

↘ Howdoestheassessmentsystemreflectthestudentcentredapproach,criticalthinking,new

didacticmethods?

↘ Whatisthepolicyregardingtheimprovementofteachingmethods?

↘ Howaretheoutcomesofthesurveyscommunicatedtostudentsandstaff?

↘ Howdo you ensure that theprogrammes are delivering?Whichmethodologies do youuse to

measurethelearningoutcomesachievement?

↘ Howdoyouensure/maximisetheengagementofthestudentsinthelearningprocess?Any

examples?

↘ Howdoyouidentifyandaddressstudentswithspecialneeds?

ESG1.4.Studentadmission,progression,recognitionandcertification

Standard 1.4 addresses the student’s life-cycle, from admission, to progression, recognition and

certification. HEIs apply consistent regulations concerning student admission, recognition and

certificationinthefourcountries.Atpresent,thedevelopmentofregulationontherecognitionofnon-

formalandinformallearningposeagreatchallengeforthedevelopmentofIQAs.Therecognitionof

progressioninjointprogrammesandforeignqualificationsarethemostcriticalissuesintheframeof

thisstandard.

In fact, considering the relevance of the standard, obstacles for implementation and further

development the topics addressed in the intercultural training events focused above all on the

institutionalmechanismsforrecognitionofpriorlearning.


↘ Whatisthebiggestproblemintheselectionofcandidates?2

nd

and3

rd

level?

↘ Whatisthepolicyoftheinstitutionforfairrecognitionofpreviouslearningachievements

?(formal,informalandnon-formalpathsmustbeconsidered)

↘ Howdoyoudealwiththeissueofrecognitionoftheprogressionofthejointprogrammes’

students?

↘ Whatkindofproceduresandmechanisms/indicatorshaveyoudeveloped/putinplaceto

successfullyevaluateandrecognisepreviousformal,non-formalandinformallearning?

↘ Howdoyouaddresstheproblemofrecognitionofforeignqualifications?

↘ Traceabilityofgraduates?Doyouidentifythebeststudents?

↘ Arethereprocessestocreateafunctionalnetworkofalumni?

↘ Whatkindofmechanismsdoyouhaveinplacetomonitorandpreventdropout?

↘ Howdoyouensureyouhavesufficientresources(human,material,financial)inorderto

guaranteethequalityofteachingandlearning?

13

ESG1.5.Teachingstaff

InthefourcountriesoftheEIQASproject,teachingstaffareassessedonaregularbasissince,insome

cases, it is required by national regulations. The remaining activities which include professional

development opportunities provided to teaching staff, incentives to encourage the professional

developmentofteachingstaff, incentivestoencouragetheuseofnewtechnologies inteaching,or

mechanismfor rewarding teachingachievementsvarygreatlyamongunitsand fieldsof study.The

mainchallengeislinkedtotheregularmonitoringofteachingstaffsatisfaction.

Some of the HEIs still report limited mechanisms for teachers support in the process of their

development;weaknessesintheareaofmotivationandassessmentoffacultycanbeobserved(there

areonly formalassessmentsorwrittenstatements,generally).Assuch,questionnairesystemsand

developmentofpedagogyforteacherscouldbedevelopedandtheregularevaluationoftheactivities

ofteachingstaffcouldbeimproved.

Somestakeholdersassertthatamanualofgoodpracticescouldbedeveloped.


↘ Whatisyourpolicyforrecruitmentanddevelopmentofteachingstaff?

↘ Howdoyoubalancetheresearch,teachingandorganisationalskillsofyourteachersandstaff?

↘ Whatarethemotivationalschemes/methodsadoptedintheHEI?

↘ Whataretheassessmentmethods/proceduresoftheperformanceoftheteachingstaff?How

aretheyconnectedwiththedevelopmentopportunities?Anyexamples?

↘ HowdoestheHEIattractforeignacademicstaff?

↘ Whataretheproceduresormechanismsforrecognitionofacademicexcellence(excellencein

teaching)?

↘ Whataretheindicatorsanddefinitionofexcellenceinteaching,research,mentoringand

motivating?

↘ Howdoyouincorporatetheresultsofstudents’questionnairesintotheSelfEvaluationReport?

↘ Howoftendoyoumonitorthestudentsatisfactionrateswithteachers/staffandwhatkindof

mechanismsdoyouhaveinplacetoperiodically/annually/sufficientlyreactininstanceswhere

youhavedetectedopportunitiesforimprovement?

↘ Howdoyoumeasure/evaluatestaffsatisfactionandhowdoyoureactininstancesofareduction

intheirsatisfaction?

ESG1.6.Learningresourcesandstudentsupport

TheimplementationofESG1.6variesgreatlyamongdifferentfieldsofstudyandunitsthroughoutthe

four countries. At present, all responding HEIs provide academic, financial and personal advice to

students.Inaddition,theyhaveinplaceamechanismforinformingstudentsaboutthesupportand

services available. At the same time, there are still HEIs which declare that they do not have

amechanismforassessingtheadequacyandaccessibilityoflearningresourcesorstudentsupport,or

donothaveinplaceprocedurestoensurethatadministrativestaffareproperlyqualifiedtodeliver

supportservices.

SomecontradictionsexistinwhatconcernsStandard1.6andmostofthestakeholdersareawareof

them.Althoughverygoodresourcesandexcellentlearningmeansexist,mostoftheHEIsdonothave

anyproceduresforassessingtheminplace.

14

Some good practices and innovative tools are available inmanyHEIs, such as rooms for students

available24haday,supportforstudentswithspecialdisabilities,tutorialprogrammesforstudents,

social support, among other infrastructures and services. There is however a need for efficient

implementation of processes covering development,monitoring, assessment and improvement of

learningresourcesandstudentssupport.

Massification and diversification of higher education increases the need for adequate learning

resourcesandstudentssupport.Increasingnumbersofstudentsaswellastheneedforequallyand

actively includingallstrataofstudents inthestudyprocessrequirespayingevermoreattentionto

studentgroupswithspecialneeds.Economicandpoliticalrestraintshowever,arestillagreatchallenge

forHEIs,inthemajorityofcases.


↘ Howdoyouassessthestudentsupportsystem,researchanddidacticinfrastructure?

↘ Whichnewtechnologieshavebeenimplementedinthetrainingprocesstoachievethelearning

outcomes?

↘ Howdoyouassessthefinancialsustainability?

↘ Whatisthelearningresourcesfundingpolicy?

↘ Whatarethelatestachievementsinimprovingthelearningresources?

↘ Whatarethedevelopmentopportunitiesforadministrativestaff?

ESG1.7.Informationmanagement

Efficient and effective internal quality assurance systems need to be supported by a reliable

informationmanagementsystem.MostHEIshaveaformalmechanismforanalysingandusingdata

collectionforqualityassuranceenhancementpurposese.g.keyperformanceindicators(KPI),profile

ofthestudentpopulation,studentprogression,successanddrop-outrates,students’satisfactionwith

programmes,etc.Theoverallanalysisofthefourcountriesshowsafairlygoodobservanceofstandard

requirementsbyHEIs;thereisgoodfeedbackbetweendataanddecisions.

Very complexandadvanced information systemsarebeingdevelopedall over Europe in theHEIs.

Business intelligent systems, automatic analysis data tools, various open source tools for data

warehouse,etc.etc.Ineachinstancetherearedifferentiatedsystemsaccordingtothenatureofthe

HEI,thedimensionofthedatabasesandtheneedforanalysis.

OneweaknessregardingtheimplementationofStandard1.7canbefoundinthelowlevel,orlackof

indicatorsofinternationalisation.


↘ Isyourinformationsystemcomprehensiveenoughtoenablegoodmanagementsupport?

↘ Howdoyouusethecollecteddatafortheenhancementofthequalityofteachingandlearning?

↘ ArestudentquestionnairesincludedintheITsystem?

↘ Whatkindofanalysisdoyoupublishinordertoassuretheaccountabilityoftheinstitution?

↘ WhatkindofdataiscollectedbytheHEI?Whatkindofanalysisisdoneandhowisitused?

↘ WhataretheHEI’sKPIs?

↘ DoestheHEIperformanymeta-analysisoftheKPIs?

↘ Arethereanypreventive-orientedanalyses?Anyanalyses?

15

↘ HowdoesIMsupportresponsibledecisionmakingandpolicymakingprocesses?

↘ Howdoyouensureuptodateinformationtosupportmoreresponsibleandefficientaction

plansandtoensuretransparencyoftheHEI’soperations?

ESG1.8.Publicinformation

HEIs in the four countries provide full information about the programmes they offer, including

admission criteria, full curricula, syllabuses etc. However, some of them declare that some legal

requirements(dataprotectionlaw)mighthinderpublicinformationactivities.Thewebsiteisthemost

commonvehicleforinformationdissemination.Agooddisseminationoftheinformationiscriticalin

anyIQAS.Itisimportanttokeepthefeedbackfromstudentsupdated.Thereisaneedfordaytoday

monitoring.

Asregardswhatconcernstheaccreditationsreportsallstakeholdersagreedthattheyshouldbefully

publishedandHEIsshoulddefineinwhichlanguage(s)officialdocumentswillbewrittenandpublished.


↘ Whatkindofdataisthereandtowhatextentisitpubliclyavailable?Inwhatform?

↘ Whatkindofpublicrelationsstrategy(includinginternationalisation)doyouhaveinplace?

↘ HowdoestheHEIassuretheactualityofinformation?

↘ Whatarethewaysofupdatingtheinformation?

↘ Howdoyouensurethatrelevantinformationisdistributedtothetargetgroups?

↘ Howdoyoulocateyourtargetgroups?

ESG1.9.On-goingmonitoringandperiodicreviewsofprogrammes

Standard1.9 is strongly linkedwithStandard1.2.A formalprocedure foron-goingmonitoringand

periodicreviewshouldexistinallIQAS.Thereviewmustbedoneregularly.Thereshouldbeanannual

reviewofprogrammeswiththeparticipationofstakeholders,includingalumniandemployers;HEIs

shouldraisetheawarenessofallstakeholdersconcerningtheimportanceoftheirparticipationinthe

process.

Infact,themostcriticalissueintheimplementationofthisstandardalloverthefourcountriesisthe

involvement of the external stakeholders in the on-going monitoring and periodic reviews of

programmes.Eveninternalstakeholdersshouldbeawareoftheimportanceoftheirinvolvement.

Anotherrelevant topic is that theevaluationmustbe linkedto the intended learningoutcomes, in

acombinationoftop-downandbottom-upapproaches.Formalandinformalapproachesshouldbe

used.InmostoftheHEIstheproceduresforon-goingreviewarepoorlyformalised.

Effectivecommunicationandprovidingfeedbacktoallstakeholdersinvolvedarecrucialtoensurethe

successful closingof the 'quality circle'. In termsofprovidingon-goingand regularmonitoringand

reviewofprogrammes, thediversityofvariousscientificdisciplinescoveredby theprogrammesat

certainHEIsshouldbetakenintoaccount.


↘ Howisthecontinuousimprovementoftheprogrammesassured?

↘ HowdoyoubuildthesystemofmonitoringtoenhanceIQAS?

↘ Howdoyouinvolvestudents?

16

↘ Howaretheexternalstakeholdersinvolvedintheperiodicevaluations?

↘ Doyouperformandanalysesurveysonstudents,employersandalumni?

↘ Arethesurveysstatisticallyrepresentative?

↘ Whyhaveyoudecidedforthisperiod,notlonger/shorter?

↘ Howaredifferentstakeholdersincludedinthemonitoringsystem?

↘ Howarethereviewsorganised?

↘ Istheworkloadoftheperiodicreviewsanissue?

↘ Howaretheneedsofsocietyidentifiedandincludedintotheprogrammes?

↘ Howdoyouchecktheadequacyofthelearningenvironmentandthesupportserviceforthe

purposeofprogrammedelivery?

ESG.1.10.Cyclicalexternalqualityassurance

GloballyallstakeholdersinvolvedintheEIQAStrainingeventassertthattheinterestintheanalysisof

EQAimpactonIQAisstillverylimitedandthatthegapreductionbetweenEQAandIQAisstillrelevant

anda challenge forHEIs andQAAs. Legislative frameworksmaydisturb theefficiencyand relation

betweenEQAandIQAprocesses.Somegoodpracticeshavebeenidentifiedsuchasorganisingpublic

presentations of audits reports and following discussions on further improvement across HEIs,

increasinginterestamongHEIsininternationalexternalqualityevaluation,andreferencetoESGinthe

nationallegislativeframework.

AccordingtotheHEIsofthefourcountriestheexternalevaluationmethodologyofqualityassurance

agenciescouldbe improvedintheaspectofthecriteriaandmethodologyofassessment,timeand

quality-relatedcosts,trainingofexpertsaswellasfurthersupportandconsultancy.Insomecountries

however,asisthecaseofPortugal,HEIsareverysatisfiedwiththeworkoftheQAA.Acomplexand

multi-areaimpactstudyontheinfluenceofEQAonIQAcouldimprovetheknowledgeoftherespective

qualityassuranceagencyandfacilitateitsadjustmenttorelevantexpectationsandneeds.

Guidingquestionsforasitevisit:

↘ TowhatextentdoestheexternalevaluationcontributetotheenhancementofyourIQAS?

↘ Wouldyouapplyforanon-mandatoryexternalevaluation?

↘ HowareEQAoutcomesintegratedintoIQA?

↘ Whatkindofmechanismsdoyouhaveinplaceforfollow-upactivities?

↘ Doyouobserve/measurethepositiveornegativeimpactofexternalqualityassurance?Doyou

observeanyimpact?Whatarethemainhindrancesorchallenges?

6. A critical analysis of the results of the workshops

In the last two decadesHEIs in the four countries of the project – Poland, Portugal, Slovenia and

Bulgaria, have witnessed a pronounced and dynamic development of IQAs. In all countries there

appeared external bodies responsible for QA. In addition, some crucial changes in law on higher

educationsystemstookplacesthat imposed, insomecases, legalrequirementstoestablishanIQA

system.Correspondingly,HEIsturnedtoamoresystematicapproachtoIQAandtotheformalisation

ofsolutionstothis issue.HEIscurrentlyhaveIQAsystemsinplacewhosefunctioningvariesgreatly

amongfaculties/unitsandfieldsofstudy.Partiallackofconsistencyandcomplexitymaybecausedby

rather low awareness of quality management principles among internal stakeholders, especially

17

teachingstaffandstudents.Qualitymanagersstillmeetinternalandexternalobstaclesindeveloping

sufficientIQAsbutatthesametimetheyprovidemanygoodexamplesofactivitiesundertakeninthat

areaandobservebeneficialqualitativeandquantitativechanges.Furthermore,frequentchangesof

legalrequirementscausesomedissatisfactionamongthoseinvolvedinIQAatmanagementaswellas

operational levels, particularly in some countries. HEIs and representatives of the different

stakeholdersdeclaredifficultiesrelatedtobureaucracywhichcanbeconsideredsomethingofa'hot

potato'inthehighereducationandqualityassurancesystematpresent.

The development of those systems and their effectiveness ismonitored on an ongoing basis and

evaluatedby theQAagencies.Allagencies frompartners’countrieswereexternally reviewed.The

continuous improvementof theexternalqualityassurancesystemsof theagencies isapriority for

theirfutureoperations.

IntermsoftherevisedESG2015,theHEIsarewillingandreadytofollowthem,inspiteoftheshort

historyinIQAsystemsdevelopmentinsomecountriesandtheirmaturity.SomeoftheHEIsthatwere

aware of beneficial changes in the qualitymanagement area aswell as the importance of quality

culture decided to introduce and develop IQA systems on their own initiative without external

pressure.Nevertheless,furtherclarificationofindividualstandardsandtheirpopularisationisgreatly

requiredsincethemajorityofstakeholdersstillidentifyinternalandexternalobstaclesinapplyingthe

revisedESG.TheresultsoftheprojectexpressaneedbyHEIsforfurthersupportinIQAenhancement

and should imply the activities at the national level (policy makers, QA agencies, conferences of

rectors)aswellasattheEuropeanlevel(ESU,EURASHE,ENQAetc.).

InwhatconcernsthecoreissuesoftheimplementationoftheESG,ashortlistcanbehighlighted.

Firstofall,allstakeholdersshouldbeawareofthefactthatthesetofstandardsisnota'checklist',i.e.

acomplianceexercisetobeperformed,thereforethemethodologyreferringtotheimplementation

ofthegivenstandardsshouldbeexplainedandwidelydiscussed.

Secondly, definition and assessment of learning outcomes; student-centred learning and mutual

recognitionofqualificationsdeliveredinotherHEIsorcountriesarestillpoorlydevelopedevenif,each

time,morediscussed.

Thirdly,theinvolvementofthestakeholders,bothinternalandexternal,isstillfarbelowthedesirable

level.Forinstance,thereissomedegreeoflowengagementandlowmotivationofstudents(incertain

casesalsoofstaff)inwhatconcernsevaluationandqualityassuranceprocedures.Sometimesthelow

level of participation is a result of a lack of understanding of the objectives. In fact, internal

communicationinsidetheHEIscorrespondsinsomecasestothedisseminationofinformationbuta

realdiscussionandcriticalanalysisdoesnotalwaystakeplace.

Fourthly,agenerallackoffundingintheHigherEducationSystemsaffectstheimplementationofthe

standards,sincetheinstitutionsareunderstaffedandlackfinancialresources.

Onbalance,wemayconcludethatpoliciesaremorecomplexthanwhatisformulatedinthewritten

documents.Theagencyofthedifferentstakeholdersisalonglearningprocess.

18

Part 2 Working arrangements which QA agencies may adopt for future exchange of experts.

ExchangeofexpertsisstillidentifiedasachallengeforQAagenciesoperatingwithinEuropeanHigher

EducationArea.EIQASproject’spartnersdecidedtofacethischallengesbytestingsomeapproachthat

mayfacilitatetheregularexchangeofexpertsandboostinternationalizationofassessmentprocedure

via involvementofforeigners. Inthischapter,theexperienceofEIQASproject issharedwithwider

audience.Itcouldbetreatedasareferencepointforfurtherpracticesinthisareaanddonothaveany

obligatorycharacter.SinceEIQASprojectfocusedonPart1ofESGandIQAsystemswetargetedat

expertsthatcanassessinternalqualityassurancepracticeswithinHEIs.

For stating the art of works within QA agencies participating in EIQAS project it was developed

a reference framework for a comparativeanalysisof agencies IQA's assessmentmethodology. The

reference framework defined an approach to comparative analysis of methodologies that the

participating QA Agencies use in assessing IQA systems at their HEIs. Each partner agency was

responsibleforprovidingcontributiontothedevelopmentofareferenceframework.Similaritiesand

differences among agencies assessment methodology was identified with the use of reference

frameworkforacomparativeanalysis.

The final report comparesassessment frameworkswith special regard to IQA systemsas currently

existingineachcountry,methodologies(assessmentprocedures,standardsandcriteria)usedbythe

Agenciesandtheirpoliciesforthepublicationofevaluationoutcomes.Itincludesacriteriareference

table showing similaritiesanddifferencesbetween theAgencies’ approaches to theassessmentof

IQAsandfinalconclusions.Aseparatesectionofthereportidentifieselementsofthemethodologyof

each Agency which are recognised by the other partners as good practice, as well as innovative

approaches or arrangements that emerged from the discussion during the seminar which are

consideredworthtransferringtoothercountries,bothwithinandoutsideofthePartnership.

Asitwasstatedin‘ComparativeanalysisofthemethodologyusedbyQAagenciesforIQAassessment’

“(…)theagencies fromBulgaria,Portugal,PolandandSloveniacoverallquality indicators fromthe

ESG,whicharerelatedto internalqualityassurance.Theseaddress internalqualitypolicies,quality

culture, transparency of internal quality assurance regulations and procedures, involvement and

consideration of stakeholders, closure of quality loops (i.e. methodological and procedural

completion),extentofqualityrelatedtopicscovered,aswellasrevisionanddevelopmentofstudy

programmes,resources,etc.Theagenciespredominantlypromotetheideologyandtheemphasesof

the ESG.” Only follow-up procedure differs considerably, however, the methodology in follow-up

proceduresissimilartothatappliedinaudits,ex-postorre-accreditationswithadifferenceinthesole

emphasis on the implementation of clearly elaborated recommendations or demands for

improvement. As you can notice from Reference Table that was elaborated in mentioned above

‘Comparativeanalysis’.

It seems that similarities identified in comparative reports overbalanced differences and laid the

foundation for further arrangement striving towards exchange of experts and enhancement of

internationalizationofassessmentprocedures.

19

ReferenceTableofSimilaritiesandDifferencesbetweentheAgencies’ApproachestotheAssessmentofIQAS2

Item PKA NEAA SQAA A3ESLegalFrameworkQualitystandardsaredefinedbylaw,i.e.generalregulations. ü ü ü ü

Quality standards are (also) defined by specific, executive

regulations.

ü ü ü ü

TheagencyusesspecificqualitystandardsthatrefertoIQA. ü ü ü ûTheexternalassessmentofIQAmostlyfocusesonqualityof

management and organization of key activities of higher

educationinstitutions.

ü ü ü ü

TheagencyfullyappliestheESGinassessingIQA. ü ü ü ü

The agency has a special audit procedure for externally

assessingIQA.

û û û ü

Agencydecisionsonaccreditationarelegallybinding. ü ü ü ü

In order to receive accreditation, the higher education

institutionsmustimplementtherequiredimprovementsand

complywithqualitystandards.

ü ü ü ü

Theagencyhasthepossibilityoflimitingaccreditationuntil

theshortcomingshavebeenremedied.

ü ü ü ü

Theagencyhasaformalappealprocedureinplace. ü ü ü ü

ProceduralframeworkInitial/ex-anteaccreditationsassessthedesignandset-upof

IQA.

ü ü ü ü

Re-accreditations,ex-postaccreditationsorauditsarebound

toassessingthealreadyfunctioningIQA.

ü ü ü ü

Item PKA NEAA SQAA A3ESThe agency considers other non-ESG systemic and

methodologicalapproachestoqualityassurance(i.e. ISOor

other).

ü ü û ü

Assessments must be based on hard evidence when

determiningcompliancewithminimumstandards.

ü ü ü ü

In decision-making, the agency considers the outcomes of

previous procedures only when following up on the

implementationofexpertoragencyrecommendations.

ü ü ü ü

The agency does not refer to other procedures, higher

educationinstitutionsortheirstudyprogrammesincontent,

butonlywhencheckingargumentative,proceduralorother

formalsolutions.

ü ü ü ü

The agency accepts internal evaluations as enhancement

proceduresandevidenceofactiveIQAprocesses.

ü ü ü ü

Theagencyconsidersthefindingsofinternalevaluationsas

evidence,whichwouldreplaceexternalevaluation.

û û û û

Inexternalassessments,theagencycarriesoutpeerreviews. ü ü ü ü

2

ComparativeanalysisoftheQAagencies’methodologiesforassessmentofIQAS,Ljubljana2015/2016,EIQAS

project.

20

External assessments may also include elements of

inspection.

û û ü û

Expertsareappointedfromagency’sinternalpoolofexperts. ü û ü ü

Higher education institutions may appeal if they disagree

withtheappointedgroupofexperts.

ü ü ü ü

Theagenciesorganizetrainingsforexperts. ü û ü ü

Theagencyhastoappointstudentexpertsaswell. ü û ü ü

Agencystaffandmembersofdecision-makingbodies steer

andharmonizetheworkofexperts.

ü ü ü ü

Audit, ex-post or re-accreditation procedures require site

visits.

ü ü ü ü

Initialorex-anteproceduresalwaysrequiresitevisits. û ü û ûInaudits,ex-postorre-accreditationprocedures,theagency

enables higher education institutions to respond to expert

reports.

ü ü ü ü

In initialorex-anteprocedures,theagencyhastosendthe

expert report to the higher education institution for

response.

ü ü û ü

An expert report is a single common report reflecting the

opinionoftheentiregroupofexperts.

ü ü ü ü

MethodologicalframeworkExpertsareexpectedtocheckforcompliancewithminimum

standards.

ü ü ü ü

Expertsareexpectedtoevaluateandgradealsotheaspects

of quality exceeding the minimum threshold, propose

recommendations for improvementandpoint to goodand

poorpractices.

ü ü ü ü

The agency encourages experts towards benchmarking or

referring to relevant national or international studies and

statisticsorresearchtobettersubstantiateandcontextualize

theiropinion.

û û û û

Item PKA NEAA SQAA A3ESThe agency uses gradingwhen externally assessing quality

and expressing the amount of compliance with quality

standardsorlevelofdevelopment.

ü ü ü ü

The agency publicly ranks higher education institutions

accordingtogrades.

û û û û

Experts may hold preparatory meetings and prepare

questionspriortositevisits.

ü ü ü ü

Whencollectingtestimoniesininterviews,expertsmaycross

examine critical information with several groups of

stakeholdersanddocumentedcontents.

ü ü ü ü

Aftertheexpertshavegatheredtheinformation,theycreate

anarrativethatmostsuitstherealityoftheassessedmatter.

They check for authenticity of gathered information and

decideonitsrelevance.Afterwards,theyputitintocontext

andthroughinterpreting,comparing,referencing,analysing,

inducing,deducing,etc.conceptualizemattersofquality.

ü ü ü ü

21

The agency is dedicated to assure that the external

assessment by experts is a process of autonomous and

impartialcriticalthinking.

ü ü ü ü

Aftertheexpertshaveestablishedtheactualstateofaffairs,

theyevaluateandgradeit.

ü ü ü ü

The agency has several regulative, organisational and

methodological instruments of achieving comparability of

externalqualityassessments.

ü ü ü ü

Theagencyisoftheopinionthattheproposedincentivesand

recommendations for improvement contribute to the

developmentofhighereducationinstitutions.

ü ü ü ü

Theagencyisoftheopinionthatexternalqualityassurance

contributestotheimpetusinenhancingqualitycultureand

progressalsofromthebottomup.

ü ü ü ü

The agency evaluates the efficiency and sufficiency of

externalqualityassessmentsinitsself-evaluationreports.

ü ü ü ü

The agency believes that its external assessment methods

lead to establishing the actual state of affairs and provide

undisputableandvalidevidenceandfindings.

ü ü ü ü

QualityindicatorsforIQAQuality indicators for IQA in particular lack clear objective

boundariesofwhatisacceptable,whatisnot,whatisgood

andwhatcouldbebetter.

ü ü ü ü

National regulations on quality assurance determine

principalpropertiesofIQA.

ü ü ü ü

Theagencyassessesaccountabilityrelatedpoliciesofhigher

educationinstitutions.

ü ü ü ü

Theagencyevaluateselementsofqualityculture. ü ü ü ü

The agency assesses transparency and applicability of IQA

regulationsathighereducationinstitutions.

ü ü ü ü

Considerationofstakeholderopinionsandincentiveswithin

IQAisanimportantqualityindicator.

ü ü ü ü

Item PKA NEAA SQAA A3ESIQA procedures should treat stakeholders equally and

provideproperimpartiality.

ü ü ü ü

The agency assessesmechanisms for approval, monitoring

andperiodicreviewofstudyprogrammes.

ü ü ü ü

External quality assessments check the extent of topics

covered in IQA andwhether all important areas of quality

havebeenaddressed.

ü ü ü ü

Theagencyexternallyassessesmethodologicalcompletionof

IQA.

ü ü ü ü

Follow-upTheagencyhasaformalfollow-upinplace. ü ü û ü

System-wideanalysesTheagencyproducessystem-wideanalyses. ü ü ü ü

Theagencyhasaspecialdepartmentforanalyticalwork. û û û ü

22

ListofabbreviationsusedinReferenceTableofSimilaritiesandDifferencesbetweentheAgencies’ApproachestotheAssessmentofIQAS:

PKA:PolskaKomisjaAkredytacyjna/PolishAccreditationCommittee,Poland

NEAA: Nacionalna Agencija za Ocenjavane i Akreditacija / National Evaluation and AccreditationAgency,Bulgaria

SQAA:NacionalnaagencijaRSzakakovostvvisokemsolstvu/SlovenianQualityAssuranceAgencyfor

HigherEducation,Slovenia

A3ES:AgênciadeAvaliaçãoeAcreditaçãodoEnsinoSuperior,Portugal

Bearingaboveinmindworkingarrangementsforexchangeofexpertsmayincludefollowingsteps:

1) In-depthanalysisofstandardsandmethodologyusedbyinterestedQAagenciesasastarting

pointofworkingarrangementsforregularexperts’exchange.AsonemayhavenoticedESG

wastreatedareferenceframeworkforsuchananalysisinEIQASprojectsinceallQAagencies

operatinginEuropeanHigherEducationAreaarerequiredtocomplywithEuropeanstandards

for quality assurance. The results of comparative analysis were disseminated on project’s

websitewithsomegeneralconclusion.

2) Common intercultural trainingsof experts targetedatexpertshiredby theQAagencies to

participate in their external evaluations. Its general purpose is to discuss the experts’

perspectiveonPart1oftheESGandEQAmethodologies,inparticulartheassessmentofIQA

aspartofEQA.Thedetailedmethodologywasdescribedinpreviouschapter.

3) Pilotexchangeofexperts(forprogrammeevaluationsinselectedfieldsofstudyorinstitutional

evaluations)thatgivesanopportunitytoshareexperienceandgainadditionalknowledgeon

theassessmentproceduresusedbynationalprovidersaswellasstrengthentheoperational

skillsofexperts.Theexchangeofexpertsshouldbebasedonbilateralunderstandingofparties

interested(e.g.bilateralagreements).Thegroupofexpertstoexchangeshouldbeagreedby

both interestedpartiesbasedoncommontrainingsresults.Thepilotexchangeprogramme

shouldbedevelopedonthebasisofmutualunderstandingofagenciestakingintoaccountthe

challenges identifiedduringprevious trainings. It shouldcoverat least the following items:

typeofexchange,objectives/learningoutcomes, requirements (professional, financial,etc.)

duration,expectedresults,feedbackfromreceivingandsendingagency.Thediscussiononthe

real effects and consequences of exchange should contribute to further development of

trainingsandexchangeframework.

The common trainings results, that was referred in previous paragraph, could be used as

aprovisionalbasis/orpre-conditionforfurtherpractical(operational)approachtotrainingson

site. It isassumedthatagroupofexpertstrainedwithintheTrainingEvent3frameworkin

January2016achieveditsresultssatisfactorilyandcouldbeconsideredaspotentialcandidates

ofpilotexchangeprogrammeorevenexpert-jobseekers.Someagenciesmaydecidetoskip

thefirststepinworkingarrangementsforexchangeofexpertsandmovetothesecondone.

4) Realexchangeofexpertsthatprovidesopportunitytohire(foreign)individualsasexpertsfor

programme evaluations in selected fields of study or institutional evaluations. It ismainly

basedontheneedsissuedbyagencyandpassedtothepartneragency.Similarly,tothepilot

exchangeofexpertstheprogramme,therealexchangeprogrammemightbearrangedwithin

the bilateral agreement framework specifying objectives, requirements (professional and

financial)aswellasitstimingandresponsibilitiesofinterestedparties(sendingandreceiving

agency). The match of needs and opportunities seems to be of utmost importance for

interestedpartiesandshouldbeidentifiedatthebeginningofexchangeprocess.Besidesitis

assumed,basedonEIQASproject results, thatpreceding trainingsofexpertsmay facilitate

further exchange by providing common ground for understanding of standards and

methodologiesused.

23

Toenhancetheefficiencyofexchange,theregularmonitoringandassessmentshouldbein

place. However, reliable assessment of exchange of experts should always relies on its

objectivesandshouldbedoneonregularbasisbybothparties.Moreover,impactofsuchan

exchangecouldbealsoconsidered inabroadersensebut itwould requireawide-analysis

approachandengagementofallstakeholders.

24

Annexes

25

Annex 1 Training Event 1 Warsaw, Poland, 26th – 30th June 2015

(Target group: representatives of Higher Education Institutions)

General description

SEMINAR ON IQA & ESG PART 1

Thefive-daySeminaronIQAandtheESGwillbeheldinWarsawon26-30June2015.Itspurposeisto

discussthepartnerAgencies’methodologiesfortheassessmentofIQA,todiscussand,wherepossible,

arriveatacommonunderstandingofPart1oftheESG,andtoidentifykeyissuesandfeaturesofthe

bestandmostinnovativepracticesinIQA.Itwillbeattendedby50participants/24individualsfrom

projectpartnersand26representativesofPolishHigherEducationInstitutions.

TrainingEvent1(SeminaronIQAandtheESG)isakeyactivitywithregardtotheachievementofthe

projectobjectives.TheEIQASprojecthastwoobjectives.Oneofthemistoincreasethecapacityof

HEIs in the participating countries to develop IQA and build a quality culture by enhancing their

awarenessandunderstandingofPart1oftheESGandidentifying,developinganddisseminatinggood

innovativepractice in internalqualityassurance.Thetrainingseminarwill includepresentationson

Part1oftheESGandIQAsystems,anddiscussionworkshopsinsmallergroupsfocussedonhoweach

aspectofPart1oftheESGcanbeinterpretedandincorporatedasgoodpracticeinIQA.TheSeminar

willbetheonlyprojecteventfortheAgenciesandHEIsspecificallydevotedtoin-depthdiscussionson

theESGandIQA.Ideasandsuggestionscollectedtherewillalsoprovidethemostsignificantinputto

theGuideto IQApromotingtheESG.Whileaproposedmethodologyforthe identificationofgood

practiceexampleshadbeendevelopedbeforetheseminar,modificationscanstillbemadeafterthe

seminartotakeintoaccountsuggestionsmadeduringworkshopdiscussions.Furthermore,although

examplesofgoodpracticeinIQAtobedisseminatedthroughtheGuidewillbeselectedmainlyonthe

basisoftheevidencealreadycollectedbytheAgenciesthroughtheirEQAprocessesandtheacademic

projectpartners’knowledgeofinstitutionalapproaches,otherexamplesmaybediscussedduringthe

seminarandthenincludedintheGuide.

AstheEIQASprojectasawholeisfinancedfromthePolishErasmus+budgettheSeminarwillbeheld

inPolandandtheseminarwillbeattendedbyalargernumberofrepresentativesofPolishHEIs.This

will increasetheimpactonPolishhighereducationintermsofenhancingtheunderstandingofthe

ESGandtheawarenessofgoodpracticearrangementsinIQA,asenvisagedintheprojectobjectives.

ThesecondobjectiveofEIQASistoincreasethecapacityoftheparticipatingQAagenciesinEQAby

comparingtheirmethodologiesfortheassessmentofinternalqualityassuranceandexchangingand

developinggoodpracticeinIQAassessment.AreferenceframeworkforacomparativeanalysisofIQA

assessment methodologies was developed prior to the seminar via online discussion/ meeting.

However,adiscussionworkshopduring theseminarwillbe theonlyopportunity fora face-to-face

discussion and an in-depth comparative analysis of the Agencies’methodologies. Conclusions will

providethebasisforacomparativereportonIQAassessmentmethodologiestobeproducedafterthe

seminar. Furthermore, in broader terms, as the seminar is planned for five days, it will give the

Agencieswhichhavenotworkedcloselytogetheryetagoodopportunitytogettoknoweachother

betterwithaviewtoinitiatingexchangeofexpertsandconcludingbilateralcooperationorrecognition

agreementsinthefuture.

26

Training Event 1 Warsaw, Poland, 26th – 30th June 2015

Programme

Day1/26thofJune/

11.00-12.00p.m. PreparatorymeetingoftheProjectManagementGroup

12.00–12.30p.m.

Welcomespeech

OverviewoftheEIQASproject

Coord.

12.30-2p.m. PresentationofPart1ESG–afterYerevancommunique

anddiscussion

2–2.30p.m. PresentationsbystudentsonkeyIQAissuesfromtheir

perspectives

ESUrepresentative

2.30–3.00p.m. Lunch

3.00-5.00p.m. PresentationoffindingsfromonlinesurveyonIQAand

theESG-Countryreports/Crosscountryreport

Report authors from

PKA,NEAA,SQAA,A3ES

Day2/27thofJune/

10.00a.m.-1p.m.

/after45

minutessessionchangeof

participants/

11.45p.m.CoffeeBreak

3parallelworkshopsinsmallergroups(howeachaspect

ofPart1ESG is interpretedbyHEIs,whichaspectsare

unclearandhowtheyareunderstoodinHEIs’practice,

where more detailed guidelines would be particularly

helpful–qualitativeinterview)ESGstandardsanalysed:

Moderators:

1.1. Policy for quality assurance and 1.10 Cyclical

externalqualityassurance

Representatives of the

partnersoftheproject

1.2. Design and approval of programmes and 1.9 On-

goingandperiodicreviewofprogrammes

2Representativesofthe


1.3.Studentcentredlearning,teachingandassessment

and1.4.Studentadmission,progression,recognitionand

certification



1.00–2.00p.m. Findingsfromworkshop,discussion

Moderators reports

frommorningsessions

2.00–3.00p.m. Lunch

3.00–4.30p.m./after45

minutessessionchangeof

participants/

2parallelworkshopsinsmallergroups(howeachaspect

ofPart1ESG is interpretedbyHEIs,whichaspectsare

unclearandhowtheyareunderstoodinHEIs’practice,

where more detailed guidelines would be particularly

helpful–qualitativeinterview)

ESGstandardsanalysed:

Moderators:

1.7Informationmanagementand1.8Publicinformation



1.5 Teaching staff and 1.6 Learning resources and

studentsupport



4.30-5.00p.m. Findingsfromworkshops,discussion

Moderators reports

fromafternoonsessions

27

Day3/28thofJune/

10.00-10.30a.m.

Presentationofthemethodologyfortheidentificationof

IQAgoodpracticeagreedby thePartnership, followed

by discussion on the definition of good practice and

methodologyfortheidentificationofgoodpractice

WG5leader

10.30-1.30p.m.

PresentationsonIQAsystemsgivenbyHEIs

PolytechnicInstituteofVianadoCastelo

UniversityofSofia

UniversityofLjubljana

UniversityofMinho



1.30-2.00p.m. Lunch

2.00-4.30p.m.

PresentationsonIQAsystemsgivenbyHEIs

UniversityofCoimbra

UniversityofMadeira

University of Social Sciences Warsaw/Lodz/Cracow/

London

UniversityofNovaGorica

UniversityofBeiraInterior



Day4/29thofJune/

10.30a.m.–1.00p.m.

A discussion workshop to compare quality assurance

agencies methodologies for the assessment of IQA

according to an agreed reference framework and to

identifygoodand transferablepractice:apresentation

ofeachAgency’smethodologyfollowedbyadiscussion

FacilitatorWG3 leader -

a representative of the


Presentations:

PKA,NEAA,SQAA,A3ES

1.00–2.00p.m. Lunch

2.00p.m.–3.30p.m.

Theworkshop for theAgencies cont.: summary of the

discussion on the Agencies’ approaches to the

assessment of IQA and the identification of good and

innovativepracticewhichmaybeusefullytransferredto

othercountrieswithinandoutsideofthePartnership

FacilitatorWG3leader-

a representative of the


3.30–4.00p.m. ESGandIQAfromteachingstaffperspective

Day5/30thofJune/

10.00-12.00a.m.

12-2p.m.

Wrap-updiscussionontheinterpretationoftheESGand

good practice; summary conclusions from each

workshopbypartners.

FinaldiscussiononthestructureofGuidetoIQA.

Facilitator: PKA

representative

28

Parallel workshops scenario 27th of June

Goals

Themaingoalsoftheworkshopsareasfollows:

1) to enrich quantitative research which was conducted as an

onlinesurveyinPart1ESG&IQAinMarch;

2) to gather opinions from participants on how each of the

standardsofPart1ESG is interpretedbyHEIs,whichaspects

are unclear and how they are understood in HEIs’ practice,

wheremoredetailedguidelineswouldbeparticularlyhelpful;

3) toprovidefurtherinputtotheGuidetoIQA.

Form

Interactive parallel sessions encouraging participants to discuss and

share their views on Part 1 ESG. The outline of themeetingwill be

focussingonpairsofstandardswhichareasfollows:

1.1. Policy for quality assurance and 1.10 Cyclical external qualityassurance;

1.2. Designandapprovalofprogrammesand1.9On-goingmonitoring

andperiodicreviewofprogrammes;

1.3. Student centred learning, teaching and assessment and 1.4.

Studentadmission,progression,recognitionandcertification;

1.7.Informationmanagementand1.8Publicinformation;

1.5.Teachingstaffand1.6Learningresourcesandstudentsupport.

Time 2-3roundsfor45minuteseachsession

Attendees

IndividualswhofilledinthesurveyquestionnaireonPart1ESG&IQA-

circa 26 representatives of HEIs (coordinators for IQA, heads of

departments,executivesetc.),mostof them fromPoland.Attendees

will be divided into 2-3 groups to give each participant a chance todiscussandbringforwardsonesview.

Moderators

2peopledealingwithonepair of standards.Oneof their tasks is to

establishadialogueaimedatmoderatingtheparticipantstodevelop

theiranswersrelatedtoaspecificfocusquestionandspecificareaof

Part 1 ESG. The other responsibility is to prepare a short report on

findingsfromworkshops.

Focusquestions

Focusquestionsrelatetothegoalsandweredevelopedcommonlyby

theprojectpartners.Asthetimeofsession isquiteshort itseemsof

utmostimportancetoreceiveanswersonthefollowingquestions:

1. Whatisthemostimportant/crucialpartofagivenstandard/or

guidelines having a potential impact on the quality

enhancement/oronthedevelopmentofqualityculture?

2. Whatarethemainobstaclesintheimplementationofagiven

standardandwhy?Aretheyderivedfromthecontentofthe

standard?

3. What are the needs for further interpretation of given

standards? Which aspect of standards are unclear forHEI's

andwhy?

4. Do you identify any good practices in relation to a given

standard?

Keyquestionstogetherwithhandoutswillbesenttoparticipantsprior

tothemeetinginordertomakediscussionmoreefficient.Additionally,

29

participantswillberequestedtowriteabouttheirrelevantneedsand

expectationsforinterpretationofeachstandard.

Additionalquestions

Furtherquestionsshouldresultfromdiscussionwithparticipantsand

willbedevelopedbymoderatorsinthecourseoftheworkshop.Their

roleshouldbesupportivetothemainquestionsratherthantocreate

otherissues.

Afterworkshop Makingthefindingsfromtheworkshopsavailabletotheothersthrough

theplenarysession–shortreportsofmoderators.

Sustainability Open box on project webpage with special attention to further

interpretationregardingneedsofPart1ESG.

30

Moderators check sheet

1. Whatisthemostimportant/crucialpartofagivenstandardhavingthepotentialimpactonthe

quality/ordevelopingqualityculturebyaHEI?

2. Whatarethemainobstacles intheimplementationofagivenstandardandwhy?Arethey

derivedfromthecontentofthestandard?

3. Whataretheneedsforfurtherinterpretationofgivenstandards?Whichaspectofstandards

areunclearforHEI'sandwhy?

4. Canyouidentifyanygoodpracticesinrelationtoagivenstandard?

Outputquestion1 Outputquestion2


31

Annex 2

Training Event 2 Ljubljana, Slovenia, 7th – 11th December 2015

(Target group: Students)

Programme

STUDENTTRAININGONIQA&ESG-PART1

Day1,Monday,7thDecember

INTERCULTURALAWARENESSWORKSHOPS:presentationsanddiscussionsonthenationalHEandQA systems, including their strengths and weaknesses; student engagement in IQA and theassessmentofIQAaspartofEQA.

11.00–12.00 Registrationofparticipants

12.00–13.00 LightLunch(tobeservedattables)

13.00–13.15 WordsofwelcomebytheDirectoroftheAgency

13.15–13.45 Lecture:TheroleofstudentsinIQAandEQA

13.45–14.15 EIQAS-Projectoutline-MaincoordinatorofEIQASProject

14.15–15.00 Lecture:ESUandStudentengagementinIQA

15.00–16.00 Lecture:BolognaprocessandglobalchallengesinHE

16.00-17.00 NationalframeworksofHEandQAsystemswithspecialattentionpaidtostrengths

and weaknesses regarding student engagement in IQA and EQA (15mins presentations made by

countryrepresentatives-Bulgaria,Poland,Portugal,Slovenia)

17.00–17.30 Wrap-updiscussionoftheday

18.00 Dinner(self-service)

Day2,Tuesday,8thDecember

DISCUSSIONWORKSHOPSONPART1OFTHEESG:ApresentationontheESG-relatedconclusionsfromTraining Event 1; each standarddiscussed from the studentperspective andmost relevantissuesidentified.KeyissuestobeincludedintheStudentGuidetoPart1oftheESGidentifiedandthestructureoftheGuideagreed.

10.00–10.40 PresentationontheESG-relatedconclusionsfromTE1

10.45–11.25 KeydifferencesbetweentheoldandrevisedESG(SQAAstaff)

11.30–12.00 Coffeebreak

12.00–13.30 Workshops:EachESGPart1standarddiscussedfromthestudentperspectiveandmostrelevantissuesidentified;DiscussiononthestructureoftheStudents'guideonIQA(Divisionofthestudentsintotwogroups,eachgroupconsistingof10students;eachgrouphasareporter).


14.35–16.30 Finalpreparationforstudyvisits(SQAAstaff)

32

16.30–17.00 Wrap-updiscussionoftheday(2reports)


Day3,Wednesday,9thDecember

VISITS by smaller multinational sub-groups to local HEIs, incl. meetings with the universityauthoritiesandstudentself-governmentbodiestoshareexperienceinIQA.KeyfindingssummarisedandalistofkeysuccessfactorsinintroducingIQAsystemsdevelopedattheendoftheday.

10.00–10.40 Divisionoftasks,determinationofthechairperson,organisationoftwogroupsofstudentsexperts(eachgroupconsistof10studentsand2staffrepresentatives)

11.00–11.15 TraveltotwoHEI’sbybus(Ljubljana,Aškerčevac.)

a)UniversityofLjubljanaFacultyofPharmacy;

b)HigherVocationalSchool,Ljubljana

11.30–14.00 2 Site visits (interviews with Management, Quality commission, the Studentrepresentatives,studentsandrandomstudents)relatingparticipationandengagementofstudentsinIQA,visittothepremises)

14.00–15.30 Lunchinthecity

15.30–16.30 AtpremisesofSlovenianQualityAssuranceAgency:Keyfindingsfromthevisits–presentations on success factors in introducing IQA systemsmade by chairpersons, exchange ofviews

16.30–17.00 TraveltoHotelandwrap-upoftheday


Day4,Thursday,10thDecember

DISCUSSIONWORKSHOPSONSTUDENTS’ENGAGEMENTINIQAANDTHEASSESSMENTOFIQAASPARTOFEQAINTHECONTEXTOFPART1ESG,involvingthestudentgroupandrepresentativesoftheSQAA:apresentationontheSlovenianapproaches(EQAprocedures,experienceandchallenges,inparticularrelatedtostudentengagementinIQAandEQA).

10.00–11.00 Slovenianapproach(EQAprocedures,experienceandchallenges)

11.00–12.00 Trainingforstudentexperts–SlovenianStudentUniontraining(representativeofSlovenianStudentsOrganisation)


13.05–16.00 Workshops

WorkshopNo.1–ExamplesofbestpracticesinengagementofstudentsinIQAandEQA

WorkshopNo.2–ChallengesrelatedtostudentsinIQAandEQA

WorkshopNo.3–InterpretationofESGPart1fromstudentsperspective

WorkshopNo.4–TheclearnessofESGPart1,impacton the IQA systems and main obstacles intheirimplementationintotheIQA

16.00–17.00 Wrap-upoftheday


33

Day5,Friday,11thDecember

Wrap-upandthewayforward:asummaryofthekeyoutcomesofthetrainingevent;adetailedplanfortheproduction,andanoutline,oftheStudentGuidetoPart1oftheESGagreed;suggestionsforpossiblefuturetrainingforstudents-expertsinmultinationalgroupscollected.

9.00–12.00 Wrap-upandthewayforward(paralleldiscussionsamongagenciesmembersandamong students on the structure of the guide, division of tasks and a detailed plan for theproduction, organising future training for students-experts, students' role in trainings, futurecooperationbetweenstudentsandagencies)


13.00–14.00 Finaldiscussion(anyotherbusiness)anddepartureofparticipants

34

Annex 3

Training Event 3 Lisboa, Portugal (A3ES), 18th – 22nd January 2016

(Target group: external experts)

TrainingEvent3(TE3)istargetedatexpertshiredbytheQAAgenciestoparticipateintheirexternal

evaluations.Itspurposeistodiscusstheexperts’perspectivesontheESGandEQAmethodologies,in

particulartheassessmentofIQAaspartofEQA.

SeveralpresentationsfromtopmanagersofQAAs,experts,teachersfromHEIsandotherguestswill

take place in order to analyse relevant case studies and compare methodologies for external

evaluation.

ThecoreofTrainingEvent3,however,istheWorkshopsandinterculturalwork.Inworkshops1,2,3

and6,eachworkinggroupwillworkseparatelyforabout90minutesandtherewillbeaplenarysession

afterwards.Workshops4and5willconsistofasimulationgameandthegroupsaregoingtoplaythe

role of a panel and of an higher education institution (HEI)meeting in a site visit, for an external

evaluationprocess,withafocuson3or4standardsfromtheESGpart1.

Theparticipants/expertsshouldworkingroupsasmuchaspossible.Thisistimeforallofusandwe

shouldfeelthatthistrainingisintensiveandbeneficialtoeveryoneinvolved.Wetriedtosetupclearly

theaimsofeachofthegroupsessionsorworkshops.Wedidnotappointachairassuch;thegroups

themselves shouldnothave too stronga chair,because itmighthinder theengagementofall the

participants. Leaving thegroup initiallyunstructured couldhave the same risks. So,wedecided to

indicate2facilitatorsorsupporterspergroupasasuggestionforstartingthework.Weexpectthat

thefacilitatorswilldescribethetaskoftheworkshopandanswerpotentialquestions.Groupsshould

howevermaketheirownarrangementsandappointachairiftheychoosetodoso;thiswouldforce

quicker ice-breaking among the participants since they will have to cooperate from the very first

minute.Workshop6willbeorganisedbyinvitationofthefacilitatorsthatwillvolunteerforeachgroup.

Duringtheworkshops,simulationgamesand'roleplay'exercises,wearegoingtousetheelectronic

platformofA3ES,formsandguidelinesforself-evaluationreports,andformsandguidelinesforthe

externalevaluationreportsofthepanels..

Eachparticipantwillreceivealoginandapasswordforaccessingtheplatform.

APortuguese institutionhas allowed theuseof its ownprocess and self-evaluation report for the

TrainingEvent.ItistheInstitutoPolitécnicodeVianadoCasteloandtheAuditprocesswascarriedout

in2012.

35


Programme

Day1/18thofJanuary/40participants

11:00-12.00H PreparatorymeetingoftheProjectManagementGroup

12:00–12:30H

Welcomespeech–WordsofwelcomebytheBoardofA3ES–

Agency for Assessment and Accreditation of Higher

Education.AsnapshotoftheAgency,creationandactivities:2009/…

12:30–14:00H Lunch-40participants

14:00-14.15H EIQAS - Project outline - Main coordinator of the EIQAS

Project

14:15-15:00H

Intercultural awarenesspresentations:national frameworks

ofHEandQAsystemswithspecialattentionpaidtostrengths

andweaknessesregardingtheassessmentofIQAS(15mins

presentationsmadebycountryrepresentative).

15:00–15:15H Coffee

15:15-15:45HSHORTTALK1:AnIntroductionto/FundamentalsofInternal

QualityAssurance.ESG2015.Presentationanddiscussionof

theIntroductiontotheEIQASGUIDE.Aproposal/draft.

15:45–17:00H

WorkingGroupsorganisationandkick-offmeetingof the4

WG; eachWG gathers a representative from each country

and guests fromPT including students. A representative of

eachcountrywillchair.Workshop1.Preparingthevisitsofday2.QUESTIONS.Questions on HEIs to be visited. IQAS design and

implementation; barriers; outcomes; pros & cons;

certification process; site visit; reports; other practical

problems.Problem1??;Problems2??,etc..

Outcome of Workshop 1: lists of problems and questions.

HowareIQASdelivering?

17:00-17:30H Wrap-updiscussionoftheday-Roadmapforthevisitsofthe

secondday

Sala02.1

Aberta

19:30h Dinner

Day2/19thofJanuary/Visitsto3HEIs

½or1day

StudyvisitstoHEIs:Visitsbysmallermultinationalsub-groups

to local HEIs, incl. meetings with the university authorities

and QA teams to share experiences of introducing IQA

systemsinthecontextofPart1ofESG.

9:30-12:30H Visit1:HEI1–10-15participants(Lunch:free)

36

9:30-12:30H Visit2:HEI2-10-15participants(Lunch:free)

8:30-17:30H Visit3:HEI3(outsideLisboa)-10-15participants(LunchattheHEI3)

19:30H/… Dinner(free)

Day3/20thofJanuary/40participants

9:30–10:30H

Working Groups work: 4 WG; each WG gathers a

representative from each country and guests from PT

including students. A representative of each country will

chair.WorkshopsondifferentapproachestointroducingIQA

andtotheassessmentofIQAaspartofEQA.Workshop 2: did you get the answers?? Positive andnegative aspects of each HEI visited. How are IQASdelivering?At9:00Htherewillbeashortpresentationontheuseofthe

electronic platform of A3ES on the ASIGQ processes

(AssessmentofIQAS)forthesupportoftheworkshopsofthe

Training Event. Every participant will receive a login and a

passwordinordertotesttheplatformandplaytheroleofan

expertintheA3ESauditprocess(simulation).

10:30–11:15HSHORTTALK2–focusonStandard1.2DesignandApprovalofProgrammes-Involvementofexternalstakeholders(With

debate).

11:15–11:30H Coffee

11:30–12:30H





andtotheassessmentofIQAaspartofEQA.Workshop3:IQASandtheStandards

12.30–14.00H Lunch–40participants

14:00–15:30H





andtotheassessmentofIQAaspartofEQA.Workshop4:Simulationgame.Playingourrole1.

15:30–15:45H Coffee

15:45–17:00H

W5. Working Groups work: 4 WG; each WG gathers a




andtotheassessmentofIQAaspartofEQA.Workshop5:Simulationgame.Playingourrole2.

37

17:00–17:30H SHORTTALK3:AcriticalviewoftheExternalAssessmentof

IQAS.PresidentoftheAgency.

19:30H Dinner

Day4/21stofJanuary/40participantsA3ESAuditProgramme

10:00–10:30H ThePortugueseSystem

10:30–11:00HThe experience of the panel of the Audit Programme in

Portugal by the Chair of the Portuguese panel for external

assessmentofIQAS.

11:00–11:15H Coffee

11:15–11:45H Questions;debate.

11:45–12:30H

SHORT TALK 4 – focus on Standard 1.3 Student-centredlearning, teaching and assessment and 1.4. – The

involvementofstudents in IQAandEQA,withspecial focus

ontheAuditPanelofA3ES(Withdebate).Presentationbythe

groupofStudentsthathaveparticipateintheTrainingEvent

2inLjubljana.

12:30–14.00H Lunch–40participants

14:00–15:30H

W6. Working Groups work: 4 WG; each WG gathers a




andtotheassessmentofIQAaspartofEQA.Workshop6 -TheMagicStickworkshop:The linkbetweenIQAS and theQAof teaching and learning.How are IQASdelivering??

15:30–15:45H Coffee

15:45–16:30H

SHORT TALK 5 – focus on Standard 1.10 Cyclical ExternalQualityAssurance–Themeta-assessmentofIQAS.Thecase

studyofamilitaryHEIsthatwentthroughanAuditprocessin

Portugal,withDebate.

16:30–17:00H Final discussion (any other business) and Farewell to guest

andtrainees.

Day5/22ndofJanuary(Partnership)

10.00-2.00H

Wrap-up discussion; summary conclusions from each

workshop by partners; discussion on arrangements for

further training of experts and exchange of experts among

theQAAgenciestobepresentedinareportproducedafter

theevent; the structureandoutline contentsof the report

agreed.

38

Parallel workshops scenario

18th of January Workshop 1. Preparing the site - visits of day 2. QUESTIONS.

Goals

Themaingoalsoftheworkshopareasfollows:1) ToprepareforthevisitsofHEIsonthe19

th

January;

2) QuestionstovisitedHEIsondifferenttopics;

3) IQASdesignandimplementation;

4) Barriers;outcomes;pros&cons;certificationprocess;

5) Sitevisit;reports;

6) Otherpracticalproblems.Problem1???;Problems2???,etc..OutcomeofWorkshop1:listsofproblemsandquestions.HowareIQAS

delivering?

Form

Interactiveparallelsessionsencouragingpeopletoengageindiscussion

and share their views on ESG Part 1 bearing in mind the following

processofexternalevaluationandESGPart2.

Time 15:45–17:00H

Attendees

Expertsfromfourcountries,stafffromfourAgenciesandotherguests

(otherexperts,coordinatorsforIQA,headsofdepartments,executives

etc.),mostofthemfromPortugal.

Attendeeswillbedividedinto4workinggroupstogiveeachparticipant

achancetodiscussandbringforwardsones’view.

Facilitators 2 people. One of their tasks is to establish a dialogue aimed at

moderatingtheparticipantstodeveloptheirquestionstotheHEIsof

day 2. Related to ESG Part 1. The other responsibility is to prepare

ashortreportonfindingsfromworkshops.

Focusquestions


theprojectpartners.Asthetimeofsession isquiteshort itseemsof

utmostimportancetoreceiveanswersonthefollowingquestions:

1. What is the most important/crucial part of the design and

implementation of an IQAS and the alignment with the

standards/or guidelines having a potential impact on the

quality enhancement/or on the development of quality

culture?

2. WhatarethemainobstaclesintheimplementationofanIQAS

andthestandardsandwhy?Aretheyderivedfromthecontent

ofthestandards?

3. What are the needs for further interpretation of the

standards? Which aspect of standards are unclear for the

expertsandwhy?


tothemeetinginordertomakediscussionmoreefficient.

Additionalquestions


will be developed by moderators during the workshop. Their role

shouldbesupportivetothemainquestionsratherthantocreateother

issues.



Sustainability TheoutputofWorkshop1shouldbeincorporatedintheGuidetoIQA

asatoolforself-evaluationandmonitoringorreviewofIQAS.

39

Facilitators sheet Participants “To Do List”

1. Topreparethesite-visitstotheHEIsonthe19

th

January; 2. QuestionstoHEIsforthevisitspurposeondifferenttopics:

3. IQASdesignandimplementation

4. Barriers;outcomes;pros&cons;certificationprocess;

5. Sitevisit;reports

6. Otherpracticalproblems.Problem1???;Problems2???,etc..



40



41


Outcome of Workshop 1: QUESTIONS TO THE HEIs

Lists of problems and questions. How are IQAS delivering?

Lisboa, 18th January 2016

WorkingGroup1(WG1)

Whichstandardseemsthemostdifficulttobefulfilled?

ESG1.1

WhatisyourframeworkforbuildingyourIQAS?

AreyoufamiliarwithESG2015?HowdoyoufinditcomparingwithESG2005?

HowisyourQApolicylinkedtothestrategicmanagementofyourinstitution?

Howareinternalandexternalstakeholdersinvolvedinthedevelopmentandimplementationofyour

IQAS?

Whatarethemainbarriers/obstaclesfortheimplementationofIQAS?

WhatimpactdoesthesocialresponsibilityofyourinstitutionhaveontheIQASyouhaveadopted?

ESG1.2.

Who is involved in designing, approving and implementing your QA procedures? What is the

procedure?

Howdoestheinstitutionrelatethelearningoutcomestothenationalqualificationframework?

Whyandwhendoyoudecidetolaunchanewprogramme?

Howaretheprogrammeslinkedtothelabourmarketneeds,socialneeds,anddevelopmentofthe

scientificdisciplines?

Whatisthesystemofidentificationofachievedlearningoutcomes(onlyteachers,externalexaminers

….)?

Howdoyouvalidatethelearningoutcomes?

ESG1.3.

Howdoyouencouragestudentstowardsactivelearning?

Howdoyoumotivatetheteacherstowardschangingtheteachingstrategy?

Howdidyouchangetheassessmentprocedures?

Inwhatwaydoestheassessmentsystemreflectthestudentcentredapproach,criticalthinking,new

didacticmethods?

Whatisthepolicyforimprovementofteachingmethods?

Howaretheoutcomesofthesurveyscommunicatedwithstudentsandstaff?

42

ESG1.4.

Whatisthebiggestproblemintheselectionofthecandidates?2

nd

and3

rd

level?

Traceabilityofgraduates?Doyouidentifythebeststudents?

Whatisthepolicyoftheinstitutionforfairrecognitionofpreviouslearningachievements?

Doyouhaveanypolicytoprevent/controldropout?

ESG1.5

Whatisyourpolicyforrecruitmentanddevelopmentofstaff?

Howdoyoubalancetheresearch,teachingandorganisationalskillsofyourteachersandstaff?

ESG1.6

Howdoyouassessthestudentsupportsystem,researchanddidacticinfrastructure?

Which new technologies have been implemented in the training process to achieve the learning

outcomes?

Howdoyouassessthefinancialsustainability?

ESG1.7

Isyourinformationsystemcomprehensiveenoughtoenablegoodmanagementsupport?

Howdoyouusethecollecteddatafortheenhancementofthequalityofteaching?

ArestudentquestionnairesincludedintheITsystem?

Whatkindofanalysisdoyoupublishinordertoensuretheaccountabilityoftheinstitution?

ESG1.8

Whatkindofdataandtowhatextentisitpubliclyavailable?Inwhatform?

ESG1.9

HowdoyoubuildthesystemofmonitoringtoenhanceIQAS?

Doyouperformandanalysesurveysonstudents,employersandalumni?

Arethesurveysstatisticallyrepresentative?

Whatisthereviewperiod?

Howaredifferentstakeholdersincludedinthemonitoringsystem?

ESG1.10

TowhatextentdoestheexternalevaluationcontributetotheenhancementofyourIQAS?

Wouldyouapplyfornomandatoryexternalevaluationaswell?LikeEQUISorsimilar?

43

WorkingGroup2(WG2)

ESG1.1.

HowistheeffectivenessofIQASassuredandmeasured?

How are external and internal stakeholders involved into the IQA processes?What are the good

practicesinthisarea?

Whataresuccessfuland/orunsuccessfulexamplesofinterventionoftheIQAS?

ESG1.2.

Howaretheexternalstakeholdersinvolvedintothedesignoftheprogramme?Anyexamples?

HowaretheLOsconceptsimplemented?

HowdoyoucheckthealignmentofILOswithNQFandEQF?

HowdoyouverifytheachievementofILOs?

ESG1.3.

Howdoyouensure/maximisetheengagementofthestudentsinthelearningprocess?Anyexamples?

Howdoyouidentifyandaddressthespecialneedsofstudents?

ESG1.4.

Howdoyoudealwiththeissueofrecognitionoftheprogressionofthejointprogrammes’students?

Howdoyouaddresstheproblemofrecognitionofforeignqualifications?

ESG1.5.

Whatarethemotivationalschemes/methodsadoptedintheHEI?

Whataretheassessmentmethods/proceduresofmeasuringtheperformanceoftheteachingstaff?

Howaretheyconnectedwithdevelopmentopportunities?Anyexamples?

HowdoestheHEIattractforeignacademicstaff?

DoestheHEIsupportdifferentcareerpathsoftheacademicstaff?

ESG1.6.

Whatisthelearningresourcesfundingpolicy?

Whatarethelatestachievementsinimprovingthelearningresources?

Whatarethedevelopmentopportunitiesforadministrativestaff?

ESG1.7.

WhatkindofdataiscollectedbytheHEI?Whatkindofanalysesaredoneandhowtheyareused?

WhataretheHEI’sKPIs?

DoestheHEIperformanymeta-analysisoftheKPIs?

Arethereanypreventive-orientedanalyses?Anyanalyses?

44

ESG1.8.

HowdoestheHEIassuretheactualityofinformation?

Whatarethewaysofupdatingtheinformation?

HowdoestheHEIverifyifthepublicinformationisuptodate?

ESG1.9.

Howarethereviewsorganised?Istheworkloadoftheperiodicreviewsanissue?

Howaretheexternalstakeholdersinvolvedintheperiodicevaluations?

Howarestudentsinvolved?

Howaretheneedsofsocietyidentifiedandincludedintotheprogrammes?

Howisthecontinuousimprovementoftheprogrammesensured?

ESG1.10.

HowareEQAoutcomesintegratedintoIQA?

WorkingGroup3(WG3)

1. System:doesitworkasasystemorasacollectionofprocedures.Isitalive?

2. ChangesaftertheimplementationoftheirIQAS:Didyouimprovesomething??

3. Meta-evaluation:doyoucarryoutsomemonitoringofyourIQASafterthecertification?

4. Impactof IQAS indifferentgroups:studentsandmanagement(questionstoaddresstothe

differentgroups)

5. Howdoesthesystemcoversthedifferentdegrees:bachelors,masterandPhD??

6. WerethereanyrecommendationswiththecertificationofA3ES??Didyouimplementthem?

Didyougotsomethingoutofit?

7. HowdotheydefineQuality??Howdotheymeasureit?WhatKPI??Business intelligence??

Dashboard??

8. ESG1.1.Thesameapproachinthe4countries.Whocreatedthemission,thevisionandthe

QAmission?Didallthestakeholdersparticipate??Students??

9. Arethestrategicgoalsinlinewiththemission?

10. DidyouusetheESGtodesignandimplementyourIQAS?IstheESGuseful??Doyouneedus?

DoesourworkontheESGhelpHEIstoimplementIQAS?

11. ESG1.7.Informationsystem:doesyoursystemwork?Doesitidentifytheproblemsandthe

reactions?

12. Involvement of internal and external stakeholders: How?? Do you have some innovative

methodologies?

45

13. HowistheSystemworkingonwhatconcernstheimprovementofteachingstaff?

14. ESG1.3.Howdoyouensurethattheprogrammesaredelivering?Whichmethodologiesdo

youusetomeasurethelearningoutcomesachievement?

15. DoyouuseanyKPI's?

16. Societysatisfaction?

17. Whatdoyouthinkaboutourquestions?Whatareyourrequestsforus,asexperts?

WorkingGroup4(WG4)

ESG1.1

How do you guarantee that all internal and external stakeholders have a common view and

understandingofyourqualityassurancesystem?

ESG1.7

HowistheinformationproducedbyyourQualityAssuranceSystem:

a) analysed

b) integrated

c) disseminated

ESG1.1/1.3/1.5

How do you incentivise teachers to bemore involved in QA procedures (esp. QA policy, Student

centredteachingandassessment)?

ESG1.5

Whatmechanismsdoyouhave inplace to support teachers’personaldevelopment (e.g. teaching,

research,teacher’sevaluation)?

ESG1.3/1.6

Howdoyouincorporatestudents'andteachers’feedbackinthecontinuousimprovementofcurricula?

ESG1.3/1.6/1.7

Howcanstudentsappraiseandinfluencechangesinthelearningprocess(e.g.assessment,teaching

methods)?

ESG1.6

Whatsystemdoyouhave inplacetoevaluatethequalityanddevelopmentneedsofyourstudent

supportservices?

46

20th of January Workshop 2: have you got the answers?

Positive and negative aspects of each HEI visited. How are IQAS delivering?

Goals

Themaingoalsofworkshopareasfollows:1) IQASdesignandimplementation;

2) Barriers;outcomes;pros&cons;certificationprocess;

3) Sitevisit;reports;

4) Otherpracticalproblems.Problem1???;Problems2???,etc..OutcomeofWorkshop1:listsofproblemsandquestions.HowareIQAS

delivering?

Form

Interactive parallel sessions encouraging participants to discuss and

sharetheirviewsonPart1ESGbearinginmindthefollowingprocess

ofexternalevaluationandPart2ofESG.

Time 9:30–10:30H

Attendees

Expertsfromthefourcountries,stafffromthefourAgenciesandother

guests (other experts, coordinators for IQA, heads of departments,

executivesetc.),mostofthemfromPortugal.Attendeeswillbedivided

into 4 groups to give each participant a chance to discuss and bring

forwardsonesview.

Facilitators 2 people. One of their tasks is to establish a dialogue aimed at

moderatingtheparticipantstodeveloptheirquestionstotheHEIsof

day 2. Related to Part 1 ESG. The other responsibility is to prepare

ashortreportonfindingsfromworkshops.

Focusquestions


theprojectpartners.Asthetimeofthesessionisquiteshortitseems

ofutmostimportancetoreceiveanswersonthefollowingquestions:

1. What is the most important/crucial part of the design and

implementation of an IQAS and the alignment with the

standards/or guidelines having a potential impact on the

quality enhancement/or on the development of quality

culture?

2. WhatarethemainobstaclesintheimplementationofanIQAS

andthestandardsandwhy?Aretheyderivedfromthecontent

ofthestandards?

3. What are the needs for further interpretation of the

standards? Which aspect of standards are unclear for the

expertsandwhy?


tothemeetinginordertomakediscussionmoreefficient.

Additionalquestions


willbedevelopedbymoderatorsinthecourseoftheworkshop.Their

roleshouldbesupportivetothemainquestionsratherthantocreate

otherissues.



Sustainability TheoutputofWorkshop1shouldbe incorporated in theGUIDEasa

toolforself-evaluationandmonitoringorreviewofIQAS.

47

Facilitators sheet Participants “To Do List”



48



49


Outcome of Workshop 2: HAVE YOUGOT THE ANSWERS? Positive and negative aspects of each HEI visited

Lisboa, 20th January 2016

WorkingGroup1(WG1)

ESGPart1 HEI1 HEI2 HEI3

1.1 1. QualityAssurancepartofstrategy;

2. Complex&formalsystem;

3. Qualityculture;

4. StructuredprogrammeQuality

ManagementBoard.

1. QualityAssuranceStrategy;

2. Involvementofext.;

3. Goodteam&culture;

4. Intheprocessofexcellence;

5. Verycomplex,inflexible

structureofQuality

Management.

1. StrongQualityCultureinacademictradition;

2. Developedproceduresconnectedtostrategy;

3. Problemofconservativeculturevs.innovative

structures.

1.2 1. Inlinewithframework(national

andEU);

2. Twoproperlyconstructed&

implementedproceduresfor

design&approvalofprogramme;

3. External&internalstakeholders

participateinQuality

Management.

1. Deepreflectionabout

national&European

framework;

2. Developedcommunication

model;

3. Properorganisational

process;

4. External&internal

stakeholdersparticipatein

QualityManagement.

1. DesignedonlearningoutcomesQFfittoEU&

nationalframework;

2. Benchmarkingprocess;

3. External&internalstakeholdersparticipatein

QualityManagement.

1.3 1. Systemdeveloped;

2. Largeparticipationofstudents;

3. Dialogue&reflectionofstudents

byquestion;

4. 70%responserate.

1. Activeparticipationof

studentunion;

2. Studentsparticipatein

Qualityprocess;

3. 80%responserate.

1. Involvementinstudentculturebystudent

union;

2. 80%responserate;

3. CommissionsTeaching&LearningPedagogy;

4. ProjectofInnovationLaboratory.

50

ESGPart1 HEI1 HEI2 HEI3

1.4 1. Monitoringofstudentscycle;

2. Designinterventionifsomething

wrong.

1. Formalisedsystemwith

reflectivepart.

1. HighlyadvancedITsystemforQM;

2. Areteachersinvolved–itisverycomplex?

1.5 1. Questionnairesystem&

developmentofpedagogyfor

teacher;

2. Prizesforbestteachers.

1. Advancedpreparationfor

courses;

2. Limitedmechanismfor

teacherssupport.

1. Strongsupportinteachingport;

2. Weakinareaofmotivation&assessmentof

faculty(onlyformal).

1.6 1. Verygoodresources;

2. NoknowledgeofQuality

Assuranceprocedures.

1.Manysupporttools; 1. Strongsupport;

2. Excellentlearningresources;

3. Constantimprovement.

1.7 1. Theyhaveacomplexandvery

advancedsystem;

2. Lackofindicatorsof

internationalisation.

1. Goodfeedbackbetweendata

&qualitydecisions.

1. ITdeveloped;

2. Verydifferentiated&complexsystem;

3. Largeamountofdata&analysis.

1.8 1. Strongfeedbackfromstudents;

2. Fullypublishedreportfor

questions.

1. Gooddissemination;

2. Daybydaymonitoring.

1. Strongfeedbackfromstudents;

2. Fullypublishedreportforquestions.

1.9 1. Formalprocedure;

2. Bottomupapproach.

1. Doneregularly;

2. Poorlyformalised.

1.Annualreviewofprogrammes.

1.10 A3ES. A3ESandInternational. A3ES.

ESGPart1+3(Portuguesesystem)

Research Applied&technical. Social&economicalsciencesmore

applied.

TypicalresearchUniversity;

(45Runits).

51

Internationalisation

10%international;

Moderatelevelofinternationalisation.

17%internationalstudents;

18programmesinEnglish;

Highlevelinternationalisation.

10%internationalstudents(trytoincrease);

Traininglearningsupport.

Relationswithcommunity

Manychannels;

Advanced.

Highinvolvement;

Manyrelationswithbusinesses.

Academictown;

CulturalroleofUniversity.

52

WorkingGroup2(WG2)

HEI1 HEI2 HEI31.1 Qualitycultureisveryvisible;

Policyofcontinuousimprovement;

ConsolidatedinformationSystem;

Involvementofstakeholders–

studentsandexternal.

Qualitycultureisveryvisibleatalllevels;

Qualityculturerelatedtothestrategicplan;


Consolidatedinformationsystem

Involvementofstakeholders–studentsand

external.

Qualitycultureisveryvisible;


Consolidatedinformationsystem;

Involvementofstakeholders–studentsand

external;

IQASisbasedonriskanalysis.

1.2

MechanismsforQArelatedto

nationallawandinstitutional

regulations;

Multi-levelapprovalofproposalsof

studycyclesofQA.

ESG;

MechanismsforQArelatedtonationallawand

institutionalregulations;

Multi-levelapprovalofproposalsofstudy

cyclesofQA;

Teachingmethodsrelevanttoachievementof

learningobjectivesandoutcomes.

ESG;

MechanismsforQArelatedtonationallawand

institutionalregulations;

Multi-levelapprovalofproposalsofstudy

cyclesofQA;

Teachingmethodsrelevanttoachievementof

learningobjectivesandoutcomes.

1.3 Participationandfeedbackof

students;

Institutionalsupportforgood

teachingpractices.

Regularpedagogicalmonitoring;

Theinstitutionstimulatesteacherstoimprove

thepedagogicalmethods.

Engagementofthestudentsinresearch

activities;

LabforTeachingandLearningInnovations.

1.4 Awardforthebestteachereach

year;

Welcomeweekwithparticipationofdifferent

stakeholders;

Awardforthebestteachereachyear;

Regularmonitoringofstudentsactivities;

Automaticcertification.

53

Institutionalmechanismsfor

recognitionofpriorlearning.

Institutionalmechanismsforrecognitionof

priorlearning.

1.5 Annualevaluationoftheactivitiesof

teachingstaff;

Manualofgoodpractices.

Annualevaluationoftheactivitiesofteaching

staff;

Awardforbestteachers-Bookofthemonth.

Annualevaluationoftheactivitiesofteaching

staff;

Biannualsurveystoteachers(andstudents).

1.6 Roomsforstudentsopen24h.

Supportforstudentswithspecialdisabilities;

Roomsforstudents;

Tutorialprogrammesforstudents.

Socialsupportforstudents;

Tutorialprogrammesforstudents.

1.7 TheISconsolidatesallthe

informationoftheQualitysystem.

Businessintelligentsystemwhichallows

integratedinformation;

Automaticanalysisofdata;

Integratedinformation.

Opensourcetoolsfordatawarehouse.

1.8 Publicinformationthroughwebsite. Publicinformationthroughwebsite. Publicinformationthroughwebsite.

1.9 Periodicrevisionofprogrammewith

participationofstakeholders–

althoughmainlyinternal.

Annualmonitoringandperiodicrevisionof

programmewithparticipationofstakeholders,

includingalumniandemployers.

Periodicrevisionofprogrammewith

participationofstakeholders–althoughmainly

internal.

1.10 Cyclical;

ExternalQA–A3ES(Eurace,Equis).

Cyclical;

ExternalQA–A3ES.

Cyclical;

ExternalQA–A3ES.

54

WorkingGroup3(WG3)

ESG HEI2 HEI3

1.1 Alignedwithstrategicplanandannualactivities

plan;

Involvementofteachersisnotcompletelyclear;

StronginvolvementofStudentUnion;

Studentsinvolvedinqualitycouncilandin

informationrecall(surveys);

Alumniinvolvedinstudentintegrationandproject

planning;

Externalconnectionisgood,particularlyinbusiness

scientificarea;

TheybuildaqualityculturewiththeISO

implementationandregularaudits–isconsidered

anintroductiontothisqualitycultureforthe

academiccommunity;inparticularforteachers;

TheypresentedaSWOTanalysisofthesystem.

Atraditionaluniversityandastrongleadership:

qualityisstrategicnotmerelyviewedasa

governmentimposition;

ISOisrelatedtosupportservicesbutalsowith

facultyprogrammes(PDCAcycle);

Veryclearqualityplanandobjectives

monetisation;

VeryclearandcompleteIQASpresentation

-mentionofthetwomoredifficult

implementationstandards:learningoutcomesand

studentcentredlearning;

Engagementofstudentsmainlyinoperational

levelbutalsoinqualitycouncil;

Fullcompliance.

1.2 Haveclearandconsolidatedprocedurefordesign

andapprovaloftheirprogrammes;

ProceduresimplementedwithAnswerswhich

havetobeansweredduringtheprocess–whyto

modify,how(includebenchmarkingwithother

programmes),Whatkindofcompetencesshould

beobtained;

55

Wellorganisedandsupportedbyanintegrated

informationsystemthatproducesindicatorsand

reportsatdifferentlevels;

Theyconsiderthefeedbackofthestudentsurvey,

UCreport,customers,andalumniemployability.

Thisstandardconnectswith1.9duetointeraction

withinternationalisationandresearch;

Themaingeneralquestionistobemoreefficient

inHowtomeasure.

1.3 Involvedindifferentdecisionstructures,andthey

havedelegatesperclassesandineachofthe

programmes;

ThestudentsunionofHEI2iscompletelyinvolved

inqualitycouncilandinmacroandmicro

processes.

Difficulttobalancetheproportionbetween

studentsgroupsandtutors;

Studentsinvolvedindifferentcommissionsand

Pedagogicalcouncil;

Incentivesforteachersfordevelopsupporting

teachinginnovationmethods;

Criteriaforassessmentispublished.

1.4 TheyhavetheYYYsystem;

Somedifficultywithdoubledegreescertification

andvalidation;

Preventiveindicatorstoreactbeforefinalresultsis

indevelopment.

AcademicmanagementSystemXXX,availablefor

students,teachers,monitoringalllifecycleofthe

students;

Thetraceabilityofstudentsandalumni:success

casesandwherearenow“thebeststudent”and

whattheydo;

Theycontrolthedropout.

1.5 I-meritusinformationsystemforteacherevaluation

integratedwithSAPthatincludesinformationof

teachingassessmentbystudents,CV’s,etc.;

Theyrelatedthisinformationtoteacherevaluation;

SomeProceduresforteachingrecruitmentbutin

internaldiscussion;

Ateachertrainingprogrammeisindevelopment;

Themostimportantquestionrelatedtoteachers:

Howproducegoodsupportteachingmaterials

56

Theyhavearegularteachingprogrammewith

internalworkshopaboutbestpracticesandtraining

innewtechnologies;

WellorganisedSupportservicestohelpteachers

andcoursecoordinatorsininformationanalysis,

reportselaboration,(academic,libraryand

informationsystemsupport).

andarticleswithpedagogicalandresearch

impact;

AtthemomentSAPisnotcompletelyintegrated

withXXX-theinterfaceisbeingdeveloped(some

informationaboutteachersisdispersed).

1.6 Workinggrouproomsavailableindifferentpartsof

HEI2(e.g.library);

Inthefirstweekoftheyeartheyhavetutorsto

supportintegration;

E-learningsupportsystem.

Theyprovidemanysocialservices;

Isafullcomplexbecauseitincludesinternational

students.

1.7 Verywellintegratedinformationsystem

withcontinuousindicatorsmonitoringsystem;

Theyhaveindevelopmentanintegrated

intelligencebusinesssystem;

Classattendancesystem.

Datawarehouse;

Excellentsystemregularlybroughtup-todate.

1.8 WebsiteinPortuguese,EnglishandChinese. Website.

1.9 Businessintelligencesystem;

Annualreportofprogrammes(basedonpedagogic

monitoring,availableonline).

Annualsurveys,self-evaluationreports;

Procedureforreviewprogrammes;

TheyhavemadeaSWOTanalysis.

1.1 TrytointegrateothercertificationswithISOand

ESGinanuniquematrix(e.g.EURACE,AMBA)

57

WorkingGroup4(WG4)

ESGP1 HEI1 HEI2 HEI31.1. 5

+cleartoall;

+integrated.

5

+matureIQAS,2ndedition;

+transparent,published.

4

+1stgeneration;

+adjustedtothefaculties,departments,decentralised;

?ifstudents,professors,staffknowthem.

1.2. 5

+good

monitoringis

basisfor

design/

revisionof

programme.

5

+comprehensivesystem,manycommitteesin

departments,schools;

+guidelinesforchangingstudyprogrammes.

4

+coordinatorforeachprogramme,responsiblealsofor

QA

+commissionforquality,newprogrammeincludeall

stakeholders;

+SWOTanalysis.

1.3. 5

+

representation

ofstudents;

+feedback

fromstudents;

+manualfor

goodpractice.

5

+clearlearningoutcomes;

+monitoring;

+feedbackfromstudents.

4

+guideforteacherswithlearningoutcomes;

+tutorsforstudents;

?implementationofguiderecommend.

-this(SCL)isthemostproblematicstandard.

1.4. 5

+noproblems,

innational

contextof

traditional

university;

+actionsfor

attracting

students;

+scholarships;

5

+goodinformationforstudentsabout

programmes(alsoforinternationalstudents,

i.e.websiteinChinese);

+trainingforsoftskills.

4

+goodstudentadministration,certificatesonline

-dependedondemographicchanges;

?admissionpolicyofuniversity.(toattractthebest,also

foreignstudents,malestudents).

58

+smalldrop

out.

1.5. 5

+supportfor

teachers,offers

fortraining;

+groupfor

monitoringof

teaching;

+competition

fornew

teachers.

5

+independentteachers;

+supportforteachertraining;

+QASmanual;

+monitoring;

+incentivesforparticipation.

5

+highrequirementfornewstaff;

+trainingprogrammeforteachers;

+evaluationofteachersbyindicators;

+studentsurveysofteachers;

+evaluationofteacher'scooperationwithsoc./com;

+managerialskillsofteachers,respondingin

informationsystems.

1.6. 4

+equipment;

+modern

spaces,new

buildings;

+3campuses;

+different

fields

-focusedon

research,not

onstudents.

5

+specialserviceforpedagogicsupport;

+servicesfor1.,2.degree,

forscholarships;

+e-learningplatform;

+disabledstudents/withspecialneeds,on-

line;

+libraryexcellent.

4

+systemfore-learning;

+tutoring;

+forstudentswithspecialneeds,international

students;

+studentadministrationonline;

+beautifuloldlibraryande-lib.&databases;

?howdoesallthiswork.

1.7. 5

+information

system

integrated,

mature;

+designof

information

systemsalso

conductedfor

5

+integrated,complete(administrative+

academic);

+teamforanalyses.

5

+developed;

+supportdecisionmaking.

59

other

universities.

1.8. 5

+public

oriented.

5

+publicationsforpublic;

+publicationsinChinese.

4

+trytobecomplete,up-to-dateinformation;

+accessibletoall;

-websiteinEnglish.

1.9. 5

+monitoring.

5

+developedinformsystem,

similartoCoimbrauniversity.

5

+meetingwithstudentseverysemester;

+discussionsindepartments;

+reports;

+monitoring.

1.10. 5

+benefitof

externalaudit.

5

+allprogrammesaccredited.

5

+helpfulrecommendationfromexternalauditors.

P1.+3Pt

1.Research +monitoring. +evaluationofeachteacher'sresearch. +usingofdatafromexternalsourcesonresearch;

+incubators,researchcentres.

2.International +partnership

with

international

universities;

+alsoin

research

projects.

+cooperationwithBrazil(students). +participationinEUresearchprogrammes;

+exchangeofprofessors.

3.Community +networkof

partnerswith

industry.

+incubators,technologyPark. +monitoringofteacher'scooperationwithcommunity;

+botanicgarden;

+universitytheatre;

+tourists,visitorsonuniversity.

60

20th of January Workshop 3: IQAS and the Standards

Goals

Themaingoalsofworkshopareasfollows:1) QASdesignandimplementation;2) Barriers;outcomes;pros&cons;certificationprocess;3) Sitevisit;reports;4) Otherpracticalproblems.Problem1???;Problems2???,etc..

OutcomeofWorkshop3…

FormInteractive parallel sessions encouraging participants to discuss andsharetheirviewsonPart1ESGbearinginmindthefollowingprocessofexternalevaluationandPart2ofESG.

Time 11:30–12:30H

Attendees

Expertsfromthefourcountries,stafffromthefourAgenciesandotherguests (other experts, coordinators for IQA, heads of departments,executivesetc.),mostofthemfromPortugal.Attendeeswillbedividedinto 4 groups to give each participant a chance to discuss and bringforwardsonesview.

Facilitators 2 people. One of their tasks is to establish a dialogue aimed atmoderatingtheparticipantstodeveloptheirquestionstotheHEIsofday 2. Related to Part 1 ESG. The other responsibility is to prepareashortreportonfindingsfromworkshops.

Focusquestions

Focusquestionsrelatetothegoalsandweredevelopedcommonlybytheprojectpartners.Asthetimeofsession isquiteshort itseemsofutmostimportancetoreceiveanswersonfollowingquestions:

1. What is the most important/crucial part of the design andimplementation of an IQAS and the alignment with thestandards/or guidelines having a potential impact on thequality enhancement/or on the development of qualityculture?

2. WhatarethemainobstaclesintheimplementationofanIQASandthestandardsandwhy?Aretheyderivedfromthecontentofthestandards?

3. What are the needs for further interpretation of thestandards? Which aspect of standards are unclear for theexpertsandwhy?

Keyquestionstogetherwithhandoutswillbesenttoparticipantspriortothemeetinginordertomakediscussionmoreefficient.

Additionalquestions

Furtherquestionsshouldresultfromdiscussionwithparticipantsandwillbedevelopedbymoderatorsinthecourseoftheworkshop.Theirroleshouldbesupportivetothemainquestionsratherthantocreateotherissues.

Afterworkshop Makingthefindingsfromtheworkshopsavailabletotheothersthroughtheplenarysession–shortreportsofmoderators.

Sustainability The output ofWorkshop 1 should be incorporated in the GUIDE asatoolforself-evaluationandmonitoringorreviewofIQAS.

61

Facilitators sheet

Participants “To Do List”



62



63

20th of January Workshop 4: Playing our role 1st round

Goals

Themaingoalsoftheworkshopareasfollows:1)Identifymethodologiesforexternalevaluation;2)Developguidelinesforexternalevaluationandsitevisits;3)ContributiontothedevelopmentofAuditManualsfortheQAAs;4)ContributiontothedevelopmentofIQASfortheHEIs.

OutcomeofWorkshop3:ContributionstotheGUIDE

Form

Interactive parallel sessions encouraging participants to discuss andsharetheirviewsontheESGbearing inmindtheprocessofexternalevaluationofIQAS.Inthe1stround,Group1willplaythePANELandwillfaceGroup2asanHEI;Group3willplaythePANELandwillmeetGroup4thatisplayingtheHEI.Group1versusGroup2willfocusonthestandards:1.1.;1.2.;1.9.and1.10.Group2versusGroup4willfocusonthestandards:1.2.;1.3.;1.4.

Time 14:00–15:30H

Attendees

Expertsfromthefourcountries,stafffromthefourAgenciesandotherguests (other experts, coordinators for IQA, heads of departments,executivesetc.),mostofthemfromPortugal.Attendeeswillbedividedinto4groupsthatwillplaytherolesofaPanelandanHEI,intworoundsofasimulationgame.

Facilitators 2 people. One of their tasks is to establish a dialogue aimed atmoderatingtheparticipantsandthesimulationgames.

FocusquestionsGroup1versusGroup2willfocusonthestandards:1.1.;1.2.;1.9.and1.10.Group2versusGroup4willfocusonthestandards:1.2.;1.3.;1.4.

Additionalquestions




64

20th of January Workshop 5: Playing our role 2nd round

Goals

Themaingoalsoftheworkshopareasfollows:1)Identifymethodologiesforexternalevaluation;2)Developguidelinesforexternalevaluationandsitevisits;3)ContributiontothedevelopmentofAuditManualsfortheQAAs;4)ContributiontothedevelopmentofIQASfortheHEIs.

OutcomeofWorkshop3:ContributionstotheGUIDE

Form

Interactive parallel sessions encouraging participants to discuss andsharetheirviewsontheESGbearing inmindtheprocessofexternalevaluationofIQAS.Inthe2ndround,Group2willplaythePANELandwillfaceGroup3asanHEI;Group4willplaythePANELandwillmeetGroup1thatisplayingtheHEI.Group2versusGroup3willfocusonthestandards:1.1.;1.5.;1.6.Group4versusGroup1willfocusonthestandards:1.2.;1.7.;1.8.

Time 15:45–17:00H

Attendees

Expertsfromthefourcountries,stafffromthefourAgenciesandotherguests (other experts, coordinators for IQA, heads of departments,executivesetc.),mostofthemfromPortugal.Attendeeswillbedividedinto4groupsthatwillplaytherolesofaPanelandanHEI,intworoundsofasimulationgame.

Facilitators 2 people. One of their tasks is to establish a dialogue aimed atmoderatingtheparticipantsandthesimulationgames.

Focusquestions Group2versusGroup3willfocusonthestandards:1.1.;1.5.;1.6.Group4versusGroup1willfocusonthestandards:1.2.;1.7.;1.8.

Additionalquestions




65

21st of January Workshop 6 -The Magic Stick workshop: The link between IQAS and

the QA of teaching and learning. How are IQAS delivering?

GoalsThemaingoalofworkshopistoanswerthequestion:

1) HowareIQASdelivering?OutcomeofWorkshop6:ContributionstotheGUIDE

FormInteractive parallel session encouraging participants to discuss andsharetheirviewsonPart1ESGbearinginmindthefollowingprocessofexternalevaluationandPart2ofESG.

Time 14:00–15:30H

Attendees

Expertsfromthefourcountries,stafffromthefourAgenciesandotherguests (other experts, coordinators for IQA, heads of departments,executivesetc.),mostofthemfromPortugal.Attendeeswillbedividedinto 4 groups to give eachparticipant a chance to discuss andbringforwardsonesview.

Facilitators 2peoplethatwillvolunteer.Theywillinvitetheothermembersofthegroupfromamongtheparticipants.

Focusquestions

Focusquestions….:1. What is the most important/crucial part of the design and

implementation of an IQAS and the alignment with thestandards/or guidelines having a potential impact on thequality enhancement/or on the development of qualityculture?

2. WhatarethemainobstaclesintheimplementationofanIQASandthestandardsandwhy?Aretheyderivedfromthecontentofthestandards?

3. What are the needs for further interpretation of thestandards? Which aspect of standards are unclear for theexpertsandwhy?

Keyquestionstogetherwithhandoutswillbesenttoparticipantspriortothemeetinginordertomakediscussionmoreefficient.

Additionalquestions




Documents

FOR THE FURTHER - European Commission...Higher Education Area (ESG) and on the intercultural awareness of internal quality assurance (IQA) with the development of common methodologies