QAC Seminar Introduction to QAC Audit

QAC SeminarQAC Seminar

Introduction to Introduction to QAC AuditQAC Audit

Trevor WebbTrevor Webb

AssistantAssistant

Secretary-GeneralSecretary-General

QAC SecretariatQAC Secretariat

HKBUHKBU12 Mar 200812 Mar 2008

Quality Assurance Council

To provide participants with:To provide participants with:• Background to the QAC; and an• Overview of the QAC audit process;• Outline of the responsibilities of audit panel members;• Introduction to the audit methodology and to the

auditing of teaching and learning;• Explanation of methods of gathering information, and

audit and interviewing techniques• Opportunity to identify issues and raise questions

Seminar ObjectivesSeminar Objectives


• QAC Structure– The Council– The Secretariat– Auditors and Audit Panels

• Primary roles of the QAC– Conduct Quality Audits of Teaching & Learning– Promote QA & QE / spread good practice

• Rationale for establishing the QAC

Overview of QAC


• Audit Manual (http://www.ugc.edu.hk/eng/qac/index.htm)

• Auditor Briefing Notes• Initial Audit Cycle• Register of Auditors• Auditor Induction & Professional

Development• CUHK and HKBU Audits

Progress To Date


QAC Quality Audits:

• will NOT question institutional autonomy• will NOT validate or re-accredit institutions

(self-accrediting status is recognised)• will NOT be excessively intrusive• will NOT be confrontational or trying to find fault • will NOT be about surprises• will NOT compare institutions• will NOT determine funding of institutions

QAC Quality Audits #1


QAC Quality Audits are:

• about self-evaluation• about peer review• about working with institutions• about partnership• about dialogue• for institutions and students• a constructive but robust process• about helping institutions to improve

QAC Quality Audits #2


Auditors are expected:

• to be familiar with the QAC’s approach/methodology• to read thoroughly the Institutional Submission (IS) and the

Supplementary Material• to provide written comments on the IS before the Initial

Meeting of the Audit Panel• to participate fully in meetings, contributing to the discussion• to avoid applying pre-conceived templates to their

consideration of the audited institution• to contribute to, and agree, the Panel’s Report• to work as a Team

(QAC Audit Manual Annex B.3)

Roles & Responsibilities of Auditors


QAC Audits are about:

Fitness for Purpose in teaching & learning, where the

Quality of Student Learning is the Centre-piece (focal

point), with an emphasis onQuality Enhancement, i.e. continuous improvement

[QAC Audit Manual Section 3.1]

QAC Approach to Audit


Audit Panels will use the ADRI methodology:

Approach (What is the institution’s purpose?)

Deployment (How does the institution achieve its purpose?)

Results (What evidence does the institution have that its purpose is being achieved?)

Improvement (What processes are in place for improvement)

in relation to 11 core Focus Areas (to be revisited later in the

Seminar)

QAC Audit Methodology & Focus Areas #1


Audit Panels examine :

• how an institution’s practices in each focus area contribute to student learning quality

• whether an institution’s objectives, procedures and outcomes are informed by appropriate external reference points

• a sample of programmes in depth, as “test cases”

(QAC Audit Manual Sections 3.3, 3.4 and 3.5)

QAC Audit Methodology & Focus Areas #2


“‘Fitness for Purpose’where institutions have different purposes which reflect their missions and role statements they have agreed with the UGC…not a review against pre-defined standards.”

The QAC Approach:The QAC Approach:


“Student learning quality is the focal point of the audit system”

“The QAC has selected a set of…activities common to all institutions, as the ‘focus areas’ of audit….taken together, the focus areas effectively define the scope of QAC audit.” (QAC Audit Manual, Section 3.4)

QAC Audit Scope & FocusQAC Audit Scope & Focus


• All HK first degree programmes and above, however funded, that are provided by UGC- funded institutions;

• Includes provision by continuing and community education arms;

• Includes self-financing and joint programmes leading to a HK award;

• Includes teaching and learning aspects of research degree programmes.

Scope of QAC Audit ActivitiesScope of QAC Audit Activities


1. Articulation of appropriate objectives2. Management planning & accountability3. Programme development and approval processes 4. Programme monitoring and review5. Curriculum design6. Programme delivery, resources, mode, learning env.7. Experiential / out of classroom learning, i.e. “external

engagement” (e.g. OS exchange, work-integrated-learning, Professional and Industry linkages, etc.)

8. Assessment9. Teaching quality and staff development10. Student participation11. Activities specific to research degrees

QAC Audit Focus AreasQAC Audit Focus Areas


The QAC Audit Process

PreparationPreparationPanel Selection (QAC)Panel Selection (QAC)

Self-Review &Self-Review &Institutional Submission (Auditee)Institutional Submission (Auditee)

Preliminary PhasePreliminary PhaseThe Initial MeetingThe Initial Meeting

Preliminary & Offshore VisitsPreliminary & Offshore Visits ((Auditors)Auditors)

The Audit VisitThe Audit Visit(The ‘main event’)(The ‘main event’)

The Audit ReportThe Audit Report (QAC)(QAC) & &Follow-Up Follow-Up (Auditee & QAC)(Auditee & QAC)

• Initiation of the process

• Nomination of additional focus areas

• Panel selection

• Self-review

• Institutional Submission

PreparationPreparation


• Self- review is “an opportunity for the institution to analyse and reflect on its objectives for student learning, on its procedures, standards and benchmarks, and…measures” (Audit Manual, Section 5.4)

• Self-review also forms the basis of the Institutional Submission to the audit

• QAC does not prescribe approach, but there are some key issues that should be addressed.

Self-review & Institutional Self-review & Institutional SubmissionSubmission


• What student learning outcomes are we trying to achieve? (graduate attributes, standards etc.)

• How are we trying to achieve our desired student learning outcomes? (policies & procedures etc.)

• How do we know we are achieving them? (what measures and benchmarks, etc.)

• How do we respond to what the evidence tells us? (what feedback mechanisms, impact on programme standards etc.)

Key Self-review QuestionsKey Self-review Questions


• QAC does not prescribe a detailed format, but….

• Auditee is expected to submit a draft Institutional Submission to QAC six months prior to Audit Visit and the final three months prior;

Two components

• Main Submission• Supplementary Materials – key documents

referred to in Main Submission.

Institutional SubmissionInstitutional Submission


• 12,000 words• Section 1 – Introduction/Structures• Section 2 – Overview of Teaching &

Learning QA System

• Section 3-N Focus Areas• Section N+1 Conclusion• References

*** Should be “Stand-alone” ***

Main SubmissionMain Submission


• Initial Meeting of the Panel (about 9 wks before AV)- Refresh on QAC audit procedures- Review of Institutional Submission to identify major

issues & questions- Selection of programmes for sample review- Identification of further information/documentation

required- Planning of Audit Visit- Decisions on whether to visit provision outside HK- Decisions on who to meet during the Audit Visit

• Preliminary Visit to Auditee to plan the Audit Visit (about six weeks beforehand)

• Other visits, such as to operations outside HK, as required.

Preliminary PhasePreliminary Phase


• Main Visit length 3-5 days

• Panel typically operates as a group, but may split for some parallel sessions

• An Audit Visit Programme and Worksheets are prepared to provide a common reference for the panel and to ensure that the necessary issues are covered.

The Audit Visit (s)The Audit Visit (s)


• Opening meeting with President/Vice-Chancellor• Staff (various groups, senior through to junior) • Students (including those taking sample programmes)• Key boards and committees• Range of stakeholders (Council Members; Employers)• Panel deliberation sessions• “Sessions on request”• “Call-back sessions”• Exit meeting

(QAC Audit Manual, section 7 and Annex A.2)

Audit Visit – Usual SessionsAudit Visit – Usual Sessions


Format:• Executive Summary• Findings for each Focus Area:

- Commendations

- Affirmations

- Recommendations• Programmes sampled (but not named)• Institutional response (Appendix)

The Audit ReportThe Audit Report


• Concern by QAC over “fairness, accuracy and balance”;

• Produced by the Audit Co-ordinator - exposure draft sent to auditee- feedback considered by AC and Chair

• Brief institutional response• Consideration by QAC• Publication• Consideration by the UGC

The Audit Report (cont.)The Audit Report (cont.)


• 18 Month Progress Report:- Action (s) taken- Person responsible- Date by which action is expected to be

completed- Means by which the institution will assess effectiveness

• Consideration by QAC• Consideration by UGC

Follow-UpFollow-Up


Any Questions?Any Questions?DiscussionDiscussion



Audit Method ComponentsAudit Method Components• Approach to auditApproach to audit (Fitness-for-purpose √) (Fitness-for-purpose √)• Audit Scope & FocusAudit Scope & Focus (Areas of Focus√) (Areas of Focus√)• Triangulation Triangulation of data & information sources of data & information sources

(Institutional Submission, supporting documents, (Institutional Submission, supporting documents, sampling of programmes, meetings sampling of programmes, meetings

• Document Analysis and Interpretation of Document Analysis and Interpretation of Information - ADRI Information - ADRI (Identification of key quality issues (Identification of key quality issues and lines of enquiry)and lines of enquiry)

• Identification and Reporting of audit findingsIdentification and Reporting of audit findings (to be (to be covered in depth in the pre-audit Intensive Workshop)covered in depth in the pre-audit Intensive Workshop)

• Auditing, interviewing, questioning, and meeting Auditing, interviewing, questioning, and meeting organisation tools & techniquesorganisation tools & techniques required for required for effective performance of the above.effective performance of the above.


II DD

RR

AAThe Critical Questions: The Critical Questions:

1.1. What What are the institution’s are the institution’s objectivesobjectives and on what and on what basis does it set these objectives? basis does it set these objectives? (A)(A)

2.2. HowHow does the institution does the institution go aboutgo about achieving its achieving its objectives? objectives? (D)(D)

3.3. How How does the institution measure does the institution measure achievementachievement of of its objectives and what its objectives and what evidenceevidence is there that the is there that the objectives are being achieved? objectives are being achieved? (R)(R)

4.4. WhatWhat processes are in place for processes are in place for improvementimprovement (and (and howhow do these processes do these processes feed backfeed back to the setting to the setting and reporting of achievement of the and reporting of achievement of the objectives)objectives)? ? (I)(I)

• How might participants be feeling on entering the meeting room?

• How can the panel help participants feel more at ease?

• How can the panel ensure that all participants are involved in the discussion?

Managing MeetingsManaging Meetings


In general, auditors should avoid:• asking complex or multiple questions up-

front (a more straight-forward question, with a follow-up question/(s) is generally more effective);

• telling anecdotes or making speeches;• detailing the situation in their own

organisation; and,• offering gratuitous advice.

Questioning Techniques #1Questioning Techniques #1


• Creating an atmosphere for genuine dialogue during the Audit Visit is extremely important

• Panel members should try to act as colleagues and peers of the participants, rather than acting as inspectors

• To this end, the questioning and discussion must be fair and polite. It must also be rigorous and incisive, as the audit report must reflect both the institution’s achievements and its areas for improvement (Commendations, Affirmations, and Recommendations)

• Panel members should ensure that all participants are engaged in the discussion, for the encouragement of an organisational culture of continuous quality improvement.

Questioning Techniques #2Questioning Techniques #2



Any Questions?Any Questions?Opportunity for Opportunity for final discussion final discussion of any Seminar of any Seminar topics.topics.


Thank youThank youfor your for your

participationparticipation

Documents

QAC Seminar Introduction to QAC Audit