Peer-Review of Student Centred Learningusing QA procedures as an instrument for changeAnthony F. Camilleri

SCL Training – 19th December 2014 Brussels.

pascl.eu

Which teacher changed your life?

Did you have a year of your life you hated because of bad teaching?

Elements of Quality Assurance

• comparison of activities with a standard

• monitoring of processes

• measurement of outcomes

• iterative improvement

• prevention of errors

pascl.eu

A look at the Quality Cycle

• Act • Plan

• Do• Check

アクション 計画

実行チェックCHECK should be read as STUDYin English Check implies ‚to hold back‘(Deming, 1980)

pascl.eu

How does QA help?

QA is a powerful toolto accelerate and systemise

change

pascl.eu

Application

• comparison of activities with a

standard

• monitoring of processes

• measurement of outcomes

• iterative improvement

• prevention of errors

use these

tools to

improve SCL

within an

Institution

pascl.eu

How we are Approaching It

Institution Applies for

QA Procedure

Completes Self

Assessment

PASCL nominate

review team

Review Team Assesses Institution

Recommendations for

Improvement

CHANGEpascl.eu

Process Requires Commitment

InstitutionalPrerequisites

pascl.eu

LEARNING FROM

PEER-REVIEW

pascl.eu

Who is a peer?

• an expert in the field• an expert in what field?• Student-Centred Learning• What is student-centred learning?

pascl.eu

Features of an Expert

experience in course delivery

familiar with best-practice

comparative perspective

sense of mission

natural communicator

pascl.eu

Features of a Reviewer

• Ethical• Open-minded• Diplomatic• Observant• Perceptive• Versatile• Tenacious• Decisive

• Self-reliant• Acting with

Fortitude• Open to

improvement• Culturally

Sensitive• Collaborative

pascl.eu

A review is about communication

University

AgencyReview Team

pascl.eu

„I don't know anything about art, but I know

what I like“

Gelett Burgess

pascl.eu

„I don't know anything about the standards, but I know quality when I see

it“

Unnamed reviewer

pascl.eu

Standards of Proof

• Some credible evidence• Preponderance of evidence• Clear and convincing evidence• Beyond reasonable doubt

pascl.eu

How to go about a Review

1. Re-familiarise yourself with the standard.2. ‘Take a Walk’ around the materials and

websites of the applicant3. Begin Review, slowly and progressively.4. Keep a Questions / Doubts pad

• Note missing evidence• Note inconsistencies

5. Check your Work

pascl.eu

William E.Deming

pascl.eu

NEVER

• leave questions unanswered• breach confidentiality• compare directly with your institution

pascl.eu

Standards of Compliance

• Substantial compliance• Partial compliance• Non-compliance

Report On:• Non-compliance• Opportunities for Improvement• Best Practice

pascl.eu

In your Report

• Be yourself

Bad: The institution showed....

Good: The review team saw / found / observed....

pascl.eu

In your Report

• Be specific

Bad: We observed a lot of....

Better: The review team found multiple and consistent examples of

pascl.eu

In your Report

• Say what you know

Bad: The institution gave us wrong information.

Better: The review team found inconsistencies between evidence (x) and interview (y)

pascl.eu

In your Report

• Give your opinion where relevant

Bad: The institution is….Bad: No Comment

Better: We suspect, It seems likely that, Given the evidence available, etc...

pascl.eu

In your Report

• Give your reasoning

Bad: There is no Quality

Better: When we consider (X), (Y) and (Z), we feel it impossible to say there is a well functioning quality cycle

pascl.eu

In your Report

• Link Effect with Cause

OK: Quality systems are in place, but there is no evidence of iterative improvement

Better: Quality systems are not effective, due to lack of procedures to mainstream improvement.

pascl.eu

In your Report

• Be Clear

OK: The physical conditions of classrooms are in need of improvement

Better: Classrooms are in a dismal state – no heating, broken desks and no boards lead to an environment which highly disincentivises learning

pascl.eu

Let’s Hear your Jokes

How to learn from self-assessment1. Read the quality standard2. Understand the quality standard3. Ask Questions4. DO NOT fill it in5. Discuss it – with all stakeholders in the institution

a. How do we meet the standard?b. What can we do better?c. Does it apply in all areas?

6. Discuss it some more – do some implementation7. Have you internalised standard?8. Now fill in the questionnaire

Be Honest

„Most Quality Managers are Magicians“

Look Here

don‘t look here!

What will self-assessment produce• A time to think, communicate and

strategize• A better understanding of institutional

processes• An overview of where we are and

hopefully where we need to go

• A report

Success Factors

• Single point of Contact• Unit or Individual Leading the Process

• Institutional Mandate / Commitment• Clear Support from Institutional Management• Inform your stakeholders

• Involvement of External Stakeholders• External viewpoint of your processes are

important

• Integration with existing Quality Tools• Map the criteria against your existing standards

Success Factors (2)

• Methods to Collect and Act on Evidence• Learning Analytics• QA Reports• Internal Development Plans

• Emphasis of Quality Improvement• how are you integrating PASCL Guidelines into

your mainstream activities?• What changes do you plan to institute?• Can you show us evidence of processes in flux?

TheShow-

Off

ThePolitici

an

TheGrump

y

TheBored

Watch out for these ppl…

How to approach a peer-review

• Understand the purpose of the review• Understand your role in the review

• Your impressions• Of your part in the instituion

• Be candid and informative• Don’t try to hide things (you’ll be caught)

Main result of peer-review

an external evaluation of your institutional quality

(procedures)

a report

a certificate

pascl.eu

Run the cycle again

• Act • Plan

• Do• Check

アクション 計画

実行チェック

pascl.eu

Remember

NO process is without errors

There is ALWAYS room for improvement

pascl.eu

What institutions should do with the report1. Read the report2. Understand the report3. Ask Questions4. Discuss it – with all stakeholders in the institution

a. What did we miss in our Self-Assessment?b. Have our assumptions been confirmed or refuted?c. How can we do it better

5. Discuss it some more – do some implementation6. Have you internalised the recommendations?7. Now start the cycle again.

pascl.eu

Principle

Only completing the circle

brings about real improvement

pascl.eu

Foundations of Auditing

• Integrity – the foundation of professionalism• Fair presentation – the obligation to report

truthfully and accurately• Due professional care – the application of

diligence and judgment in auditing• Independence - the basis for the impartiality of

the audit and objectivity of the audit conclusions

• Evidence-based approach - the rational method for reaching reliable and reproducible audit conclusions in a systematic audit process

ISO 19011:2001

pascl.eu

Under the following conditions:

Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

Share Alike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original..

Released under a Creative Commons Attribution ShareAlike 4.0 International License

You are free:to Share — copy and redistribute the material in any medium or formatto Remix — remix, transform, and build upon the materialfor any purpose, even commercially.

This project has been funded with support from the European Commission. This publication reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

pascl.eu

THANK-YOUfor your attention

Feedback is welcome!

Anthony F. Camillerianthony@camilleri.comhttp://www.slideshare.net/anthonycamilleri/