Upload
melissa-byrd
View
215
Download
0
Embed Size (px)
Citation preview
György BAZSA President, HAC
ENQADebriefing/brainstorming meeting for the
training of experts for agency reviews
Barcelona, 17 March 2009
A possible option: grading scheme for judging
compliance with criteria
Basic quantitative requirements - of a fine wine:
ethanol 9 – 13 %sugar 0 – 100 g/lpH 3 – 4
- of a nice hostess:bust size 80 – 99 cmhip size 60 – 65 cmwaistline 75 – 90 cm
- of an excellent chemist:publications minimum 50citations minimum 200impact factor minimum 40
But still there is a need for the evaluation of its/her/his quality by experts!
Now, what about a good quality assurance agency?
i. Membership
REGULATIONS OF ENQA(as from 26 September 2008)
TITLE III. INVOLVEMENT IN ENQA
a) Full MembershipFull Membership of ENQA is open to quality assurance
agencies that meet the relevant (‘eight’) criteria de-scribed in Annex I Membership Provisions document.
b) Candidate MembershipCandidate Membership is available to quality assurance
agencies that meet the criteria described in Annex I Membership Provisions document. (i.e. ‘substantial compliance with a majority of the criteria”)
ANNEX 1 – Membership Provisions
CHAPTER I. CRITERIA FOR FULL MEMBERSHIP
a) Full Membership of ENQA is open to quality assu- rance agencies in the field of higher education from EHEA member states that have been operating and conducting actual evaluation activities for at least two years.
Before being accepted as a Full Member, an applicant agency must satisfy the Board that it meets the eight criteria, listed below. The applicant agency will thereby also meet the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). - (8 or 15 criteria?)
b) An applicant agency may apply for Candidate Membership rather than Full Membership in the first instance. The Board will grant such member-ship if it believes
- that the applicant demonstrates, through its application, substantial compliance with a majority of the criteria and further believes
- that the applicant will be able to demonstrate, through an acceptable external review submitted within no more than two years Candidate Membership, full compliance with the criteria for Full Membership.
QAAdocuments QAAexpert report
Candidate2y m. FullFull5y5y member member
ENQA
ENQA
non non
non non
ii. Evaluation of the compliances by the expert panel
Dictionary and stages of compliance
Compliant: in accordance with a specified body or rules (usually used in combination):
Fully (compliant): entirely or wholly
Substantially (compliant): of ample or considerable amount, quantity, size, etc.
Partially or partly (compliant): in part, to some extent or degree, not wholly; being such in part only, not total or general, incomplete
Non-compliant: failure or refusal to comply, as with a law, regulation, or term of a contract.
ENQA National reviews (quide):
summarised conclusion stating whether the agency is
- fully or substantially compliant;
- partially compliant; or
- non-compliant.
ESU
The study visit team believes that objectivity is best served by using a three level marking system.
- Compliance, where QA implements the ESG in an effective manner;
- Partial compliance, where QA practices are based on the correct interpretation of the ESG, positive aspects have been identified but the manner of implementation is not effective enough.
- No compliance, where ARACIS practices are based on misinterpretations of the ESG …;
EUA
- fully compliant
- substantially compliant
- partly compliant
- (non-compliant)
Summary (in the case of ARACIS): substantially compliant
March 2009!: Guidelines for external reviews of ENQA member and applicant agencies- fully compliant: the Agency adheres to the letter
(literal interpretation) and spirit (intent) of thecriterion/ESG standard;
- substantially compliant: if the criterion/ESGstandard is not fulfilled to the letter, the Agency might still be considered substantially compliant if the principle/spirit of the stipulated criterion/ESG standard is followed in practice;
- partially compliant: the Agency does not fully or substantially comply with the criterion/ESG standard;
- non-compliant: the Agency does not comply with the criterion/ESG standard.
The way in which the membership criteria/ESG are to be interpreted by the review panel and the Board
A possible option: grading scheme for judging compliance with different criteria
EvaluationEvaluation GradingGrading
FullyFully compliant compliant + + ++ + +
SubstantiallySubstantially compliant compliant + + / -+ + / -
PartiallyPartially compliant compliant + / - - + / - -
NonNon-compliant-compliant / / - - -- - -
The grading is proposed by the review panel
„The Review panel will need to exercise its judgement, in the light of all the evidence before it, and the intentions of the ENQA membership criteria / ESG, as to whether substantial compliance is being achieved, taking account of the circumstances in which the Agency is required to operate (see section 3.6.2). There is little point in adopting a ‘hard line’ position in respect of compliance with the ESG if, by doing so, trustworthy and credible agencies are prevented from gaining Full membership of ENQA through no fault of their own.”
„There may, as a result, be occasions where a strict reading of the ENQA membership criteria / ESG would indicate substantial rather than full compliance with one or more standards, but the panel may come to the conclusion that this should not jeopardise an overall judgement of compliance. This may occur when national legislation requires a different approach, or where an alternative way of achieving an objective is used, or because of other justifiable reasons.”
„Where an agency is found to be either partially compliant (+ - -) or non-compliant (- - -) with a criterion, the reason for this should be explained. Full or substantial compliance may be impossible for some agencies, owing to restrictions placed on them by the very nature of their work and/or legislation in place in their country(ies) of operation. When considering such cases, the ENQA Board will take mitigating circumstances such as these into account.”
A possible option for 15 criteria: grading scale for granting ENQA membership
Decided by the BoardProposed by the review panel
PointsPoints LimitLimit Overall Overall grading?grading?
Membership Membership grantedgranted
45 – 4145 – 411 partly c.1 partly c.
0 non-c.0 non-c.Fully ?Fully ? Full memberFull member
40-39-3840-39-38 Board decidesBoard decides
37-36-3537-36-352 partly c.2 partly c.
1 non-c.1 non-c.Substan-Substan-
tially ?tially ?Candidate Candidate membermember
34-33-3234-33-32 Board decidesBoard decides
31 31 →→ ------ Non Non Non-memberNon-member
PointFully 3+
Subst 2+
Partly 1
Non 0
Sum
Full member
41
12 2 1 0 41
11 4 0 0 41
41
13 1 0 1 !!! 41 13 0 2 !!! 0 41
Full or candidate member
403938
11 1 3 0 38
10 3 2 0 38
9 5 1 0 38
8 7 0 0 38
Candidate member
373635
11 1 1 2 36
10 2 2 1 36
9 4 1 1 36
8 5 2 0 36
7 8 !!! 0 0 37
PointFully 3+
Subst 2+
Partly 1
Non 0
Sum
Candidate or non-member
343332
10 0 2 3 !!! 32
9 1 3 2 32
8 3 2 2 32
8 2 4 1 32
7 5 1 2 32
7 3 5 0 32
6 6 2 0 32
Non-member31
8 3 1 ! 3 ! 31
8 2 3 ! 2 ! 31
8 1 5 ! 1 ! 31
iii. Further proposals
- renumbering of ENQA criteria
- public list of data
- publicity of ENQA evaluation
- QA vs. QP
1 1 Executive summary 2+ - Glossary3 2 Introduction (giving contextual information: reason(s) for the commis-
sioning of the review; the place of the agency in the quality assurance structure of its jurisdiction; the main functions; the engagement of the agency with the ESG);
4 3 Findings (each membership criterion should be discussed separately)4.1 3.1 ENQA criterion 11 1: Activities / ESG 3.1, 3.34.2 3.2 ENQA criterion 11 2: External qual. assurance proc./ESG 2 4.3 3.3 ENQA criterion 22 3: Official status / ESG 3.24.4 3.4 ENQA criterion 33 4: Resources / ESG 3.4 4.5 3.5 ENQA criterion 44 5: Mission statement / ESG 3.5 4.6 3.6 ENQA criterion 55 6: Independence / ESG 3.6
4.7 3.7ENQA criterion 66 7: External quality assurance criteria and processes used by the members / ESG 3.7
4.8 3.8 ENQA Criterion 77 8: Accountability procedures / ESG 3.8 5+ 4 Any sections relating to additional Terms of Reference of the review6 5 Conclusion and recommendations (?)7 6 Annexes (Key pieces of evidence – i.e. extracts from legislation,
policies and procedures etc. – may be added as appendices)
Further proposals (out of scope of the title):
ENQA should prepare a list of compulsory data and information about QAA to be posted permanently on member agency’s homepage (partly in English).
Application, self-evaluation report, date of site visit and composition of visiting panel should be posted on the QAA’s homepage like on ENQA homepage.
QA vs. QP: quality assurance is crucial, but qua-lity of the „product” (learning outcomes etc.) is the main aim. Should we move to QP direction?
Thank you for your kind attention!
ENQA Board decisions: (see Scrutiny form)
3. Full members’ mandatory cyclical review in each 5 year:
- full membership re-confirmed for another 5 years or
- candidate membership granted in order to conform with full membership criteria in 2 years or
- neither full, nor candidate membership is granted?