Assessment validation - vetinfonet.dtwd.wa.gov.auvetinfonet.dtwd.wa.gov.au/tpf/Documents/2015 resources/Renae... · Assessment validation ... 2015’ or visit the VET Regulator websites;

Assessment validation

Renae Guthridge WA Training Institute

This session will also be webcast

The views and resources presented in this workshop represent the views

of the presenter(s) and do not in any way represent the views of VET

Regulators.

For further information on the Standards or matters of VET compliance,

please consult the ‘Standards for Registered Training Organisations

(RTOs) 2015’ or visit the VET Regulator websites;

Training Accreditation Council www.tac.wa.gov.au;

Australian Skills Quality Authority (ASQA) www.asqa.gov.au.

http://www.comlaw.gov.au/Details/F2014L01377/Download

http://www.comlaw.gov.au/Details/F2014L01377/Download

http://www.tac.wa.gov.au/

http://www.asqa.gov.au/

What is your understanding of validation?

Validation Q&A

http://bit.ly/1Et63yK

Validation is the quality review of the assessment process.

…checking whether assessment tool/s produce/s valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the training package or VET accredited courses are met.

It includes reviewing a statistically valid sample of the assessments and making recommendations for future improvements to the assessment tool, process and/or outcomes and acting upon such recommendations.

Standards for RTOS (2015) – Validation Definition

Validation Purposes Include

To ensure our assessment judgements are consistent

Increase assessor confidence

Determine whether different assessors using the same tools collect the same types and levels of evidence

Provide professional development

Address any identified area of risk in assessment practice and quality

Determine whether assessment decisions reflect the principles of assessment and rules of evidence.

What Documentation is Needed?

Instructions for Candidate

Instructions for Assessor

Mapping Documents

Assessment Strategies & Plan

Marking Guide

Instruments Evidence – direct, indirect & supplementary

Who wants Quality Assurance? Industry / Employers

Technical / Subject Matter Experts

Learners / Candidates

Assessors / Trainers

Registering / Course Accreditation Bodies

What are our Benchmarks of Quality? Standards for RTOs

AQF

Principles of Assessment

TAE10 Training and Education Training Package…assessment units.

Training Packages / Accredited Courses

Rules of Evidence

What Aspects of Assessment Process Need Validating?

Planning assessment processes (industry consultation, links with delivery, TP/AC requirements, recognition of client needs)

Delivery of assessment services (modes, plans, resources)

Processes for the review of assessment services (validation and continuous improvement)

Judgement making procedures

Evidence gathering processes and tools (on-the-job, workplace, simulation)

Support for learners / candidates (Information, feedback, flexibility, reasonable adjustment, appeals)

Management of assessors / evidence gatherers (selection, supervision, professional development)

How? Assessors working in collaboration to review, compare and evaluate assessment processes and outcomes

It’s systematically planned and implemented

Verifying consistency of evidence and judgements made against benchmark

Foxes and Vixen

Can occur before, during and after assessment

Validating methods, tools and interpretation of evidence

Foxes and a Vixen

Principles of Assessment Rules of Evidence

Four

Very

Rampant

Foxes

A

Satisfied

Cunning

Vixen

Foxes and a Vixen

Principles of Assessment Rules of Evidence

Flexibility

Validity

Reliability

Fairness

Authenticity

Sufficiency

Currency

Validity

Standards for RTOs 2015

Standard 1 – The RTO’s training and assessment strategies and practices are responsive to industry and learner needs and meet the requirements of training packages and accredited courses • comprehensive plan of systematic validation • risk-based approach to developing the plan • Trainers and assessors involved in training and assessment may also

participate in the validation process 1.9. The RTO implements a plan for ongoing systematic validation…for each training product on the RTO’s scope of registration: a) when assessment validation will occur; b) which training products will be the focus of the validation; c) who will lead and participate in validation activities; and d) how the outcomes…will be documented and acted upon.


1.10. …each training product is validated at least once every five years, with at least 50% of products validated within the first three years of each five year cycle, taking into account… risks…on the RTO’s scope…including those risks identified by the VET Regulator. 1.11. …systematic validation…is undertaken by one or more persons who are not directly involved in the particular instance of delivery and assessment of the training product being validated, and who collectively have: a) vocational competencies and current industry skills relevant to the assessment being validated; b) current knowledge and skills in vocational teaching and learning; and c) the training and assessment qualification or assessor skill set referred to in Item 1 or 3 of Schedule 1. Industry experts may be involved in validation to ensure there is the combination of expertise set out in (a) to (c) above.


Independent validation of training and assessment qualifications 1.25. From 1 January 2016, to deliver any AQF qualification or assessor skill set from the Training and Education Training Package (or its successor), the RTO must have undergone an independent validation… Independent validation means the validation is carried out by a validator/s who: a) are not employed or subcontracted by the RTO to provide training and assessment; and b) have no other involvement or interest in the operations of the RTO.

Standard 2. The operations of the RTO are quality assured. 2.2. The RTO: b) systematically evaluates and uses the outcomes of the evaluations to continually improve the RTO’s training and assessment strategies and practices. Evaluation information includes but is not limited… validation outcomes...


Users’ Guide Standards for Registered Training Organisations (2015) – TAC, ASQA

Fact sheet—Conducting validation inc Validation Sample Size Calculator – ASQA

TAC refers to - http://www.raosoft.com/samplesize.html

More Information…


What do you think? What impact is this going to have on consistency of assessment? What will need to change in your RTO?

A sample assessment process worth considering…

Ensuring Validity -

Unpacking the Unit of Competence

Mandatory unit requirements include: • Pre-requisites and/or co-

requisites • Elements • Required Skills • Required Knowledge • Range Statement (‘must’) • Critical Aspects of

Evidence • Other “must”

requirements

Mandatory unit requirements include:

• Pre-requisite(s)

• Elements

• Foundation Skills

• Performance Evidence

• Knowledge Evidence

These can be listed in a Unit Requirements List…

Unit Requirements List

Unit of competency: SHOP123A Review stock

Mandatory unit requirements

Elements of competency

1. Identify stock requirements

2. List stock in store

3. List stock shortages

4. Prepare stock order forms

Foundation skills

1. Literacy skills (reading and writing)

2. Numeracy skills

Performance evidence

1. Accurate list of stock on display

2. Accurate list of stock in store-room

3. Accurate list of stock to order

4. Two categories of stock

Knowledge evidence

1. Organisation procedures

2. Range of stock management processes

Pre-requisites: SHOP101A


Mandatory unit requirements






Range Statement (musts)

1. Stock on display

2. Stock in storeroom

Critical Aspects of Evidence

CAE 1 Accuracy and timeliness

CAE 2 Full review of stock

Required Skills

RS 1 Communicates with staff

RS 2 Correct forms selected

RS 3 Correct calculations

Required Knowledge (See list) Pre-requisites: SHOP101A

Other (specify): Two contexts

Old Unit Structure New Unit Structure

Ensuring Validity - Identifying Mandatory Unit Requirements

• Highlight all mandatory requirements in a unit of competency.

• Prepare a list of all the mandatory requirements. • Validate your list with a colleague by comparing lists.

• Prepare a flexible and fair evidence-gathering plan.

• Validate your plan with a colleague.

Planning to Gather Evidence

The Unit Requirements List can now be extended to show the relationship between the unit requirements and evidence sources.


Mandatory unit requirements Sources of evidence

WP1 WP2 SIM1 SIM2 Q/A






Range Statement

1. Stock on display

2. Stock in storeroom




Required Skills




Required Knowledge (see list)

Pre-requisites: SHOP101A

Other (specify): Two contexts

WP1: Context of workplace assessment

WP2: Context of workplace assessment

SIM1: Context of simulation

SIM2: Context of simulation

Assessment Planning Matrix

Shade the cells that don’t apply.

The blank cells in a column provide a space to record evidence later on when it has been gathered.

Developing Evidence-gathering Tools - Assessing Required Knowledge / Knowledge Evidence

Assessing ‘know-how’…

…tell me…

…written or oral short answer tests.

Ensuring Validity, Reliability and Sufficiency - Assessing Required Knowledge / Knowledge Evidence

Is the AQF level of the questions appropriate? Is the context and content of the questions relevant to the unit? Do the questions extend beyond the elements and demonstrate transfer? Is ALL required knowledge covered? If other knowledge is covered, has it been distinguished from the required knowledge? Is ALL required knowledge demonstrated before signing off on the unit record?

Developing Evidence-Gathering Tools - Assessing Elements and Required Skills / Performance Evidence

Assessing can do…

…show me…

…observation in the workplace or a simulation of a workplace.

Unit of competency: SHOP123A Review stock Name of student: Joe Bloggs

Mandatory unit requirements Performance Criteria Comments Result

E1. Identify stock requirements 1.1 Contextualised for third party

observer.

The observer would write a narrative

of what was observed for the assessor

to review. 1.2

1.3

1.4

E2. List stock in store 2.1

2.2

2.3

RK1 Stock on display Display stock not reviewed N/a

RK2 Stock in storeroom

CAE1 Accuracy and timeliness Review completed to company standards

CAE2 Full review of stock All storeroom stock reviewed

RS1 Communicates with staff

RS2 Correct forms selected

Assessment context: Two different contexts required Frozen foods

Observer’s comments:

Observer’s name: M. Ployer Observer’s position: Supervisor

Signature and date: M. Ployer 5/8/10 Student signature and date: Joe Bloggs 5/8/10

Workplace Observation Tool 1 (WP1) (3rd Party Observer)



E3. List stock shortages 3.1

3.2

E4. Prepare stock order forms 4.1

4.2

4.3

RK1 Stock on display

RK2 Stock in storeroom Storeroom stock not reviewed N/a

CAE1 Accuracy and timeliness Met our company standards

CAE2 Full review of stock Full range of display stock reviewed

RS1 Communicates with staff Did not ask for advice or report problems to supervisor


RS3 Correct calculations

Assessment context:

Two different contexts required Dry goods


Observer’s name: Sam Smith Observer’s position: RTO Assessor

Signature and date: Sam Smith 7/9/10 Student signature and date: Joe Bloggs 7/9/10

Workplace Observation Tool 2 (WP2) (RTO Observer)



E1. Identify stock requirements 1.1

1.2

1.3

1.4

E2. List stock in store 2.1

2.2

2.3

E3. List stock shortages 3.1 Incorrect stock list used.

3.2

E4. Prepare stock order forms 4.1

4.2

4.3

RK1 Stock on display

RK2 Stock in storeroom N/a

CAE1 Accuracy and timeliness

CAE2 Full review of stock


RS3 Correct calculations

Assessment context: Two different contexts required Dry goods




Simulation Observation Tool (SIM)


Required knowledge

items Questions Verbatim answers or comments Correct Score Criterion Result

RK1. 1 2 2

3

RK2. 2

2 3 6

Hot water is mixed with soap powder to make a warm

nourishing drink

7

8

RK3. 4 1 1

RK4. 5

4 3

9

10

11

13 Record the incorrect answer here ….

RK5. 12 1 1

RK6. 14 2 2

15

RK7. 16

2 2 17

18

Overall unit result

Extra knowledge Questions

EK1. 19

EK2. 20

Sandals can be worn in the cool room

EK3. 21

EK4. 22



Question & Answer Tool (Q/A)

Ensuring Validity, Reliability and Sufficiency - Assessing Elements and Required Skills / Performance Evidence

Are clear instructions provided to the observer? Are clear instructions provided to the candidate?

Are all four dimensions of competency (task skills, task management skills, contingency management skills, and job/role environment skills) met?

If third-party evidence-gatherers are used, have the performance criteria been contextualised by writing them in familiar language (plain English) and with examples, and has space been provided for narrative to explain the or assigned to each criterion?

Recording evidence

Use the Assessment Planning Matrix as a record of individual evidence by adding the name of the student and a ‘Result’ column and recording evidence from the tests and observation checklists into the blank cells.



WP1 WP2 SIM1 SIM2 Q/A






Range Statement

1. Stock on display N/a

2. Stock in storeroom N/a N/a N/a




Required Skills




Required Knowledge (See list)

Pre-requisites: SHOP101A Pre-requisite unit achieved in 2007 at XYZ College

Other (specify): Two contexts Frozen foods Dry goods Dry goods Dry goods

WP1: Frozen foods in storeroom

WP2: Dry goods on display

SIM1: Dry goods on display

SIM2: Supplementary assessment

Name of assessor: Sam Smith Student signature and date:

Signature and date: Result reported date:

Assessment Record

Meeting the Rules of Evidence - Recording evidence of competency

Are checklists used to record evidence as it happens?

Is poor quality evidence discarded?

Is evidence gathered and collated from checklists?

Is evidence from different units in a cluster distributed to appropriate unit records?

Making the Assessment Judgement

The Assessment Record can now become a

Judgement Tool showing which

requirements had been met in the

‘Results’ column.



Result WP1 WP2 SIM1 SIM2 Q/A






Range Statement

1. Stock on display N/a

2. Stock in storeroom N/a N/a N/a




Required Skills




Required Knowledge (See list)

Pre-requisites: SHOP101A Pre-requisite unit achieved in 2007 at XYZ College

Other (specify): Two contexts Frozen foods Dry goods Dry goods Dry goods

Result





Name of assessor: Sam Smith Student signature and date: Joe Bloggs 6/11/15 Signature and date: Sam Smith 6/11/15 Result reported date: 14/11/15

Judgement Tool

Meeting the Foxes and Vixen - Making the assessment judgement

Has sufficient evidence been gathered for each mandatory requirement? Is the evidence authentic and current? Have alternative arrangements been offered for candidates with specific needs? Has each mandatory requirement been demonstrated? Has the record been signed off by person(s) with assessor qualifications and vocational expertise? Have other uses of the record been quarantined from the competency judgement?

Training and Assessment Plan

The Assessment Planning Matrix, Assessment Record and Judgement Tool can now morph into a Training Delivery and Assessment Plan by listing the learning resources for each unit (and any extra) requirements.


Mandatory unit requirements Learning resources Sources of evidence

Result Session Reference WP1 WP2 SIM1 SIM2 Q/A


1. Identify stock requirements 1 - 3 pp 15 - 23

2. List stock in store 4 pp 37 - 39

3. List stock shortages 5 - 7 Handout 1

4. Prepare stock order forms 9 Handout 3

Range Statement

1. Stock on display 8 Handout 2 N/a

2. Stock in storeroom 8 Handout 2 N/a N/a N/a


CAE 1 Accuracy and timeliness 4 p. 38

CAE 2 Full review of stock 4 p. 39

Required Skills

RS 1 Communicates with staff 7 Handout 1

RS 2 Correct forms selected 8 Handout 2

RS 3 Correct calculations 7 Handout 1

Required Knowledge (See list) 1 - 9 All

Pre-requisites: SHOP101A Student records Pre-requisite unit achieved in 2007 at XYZ College

Other (specify): Two contexts 1 - 9 All Frozen foods Dry goods Dry goods Dry goods

Result





Workplace or regulatory requirements:

OH&S: Use of steps & cool room access

Name of assessor (Assessment): Sam Smith Name of Assessor (Vocational): Sam Smith Student signature & date: Joe Bloggs 6/11/15

Signature and date: Sam Smith 6/11/15 Signature and date: Sam Smith 6/11/15 Result reported date: 14/11/15

Training and Assessment Record

Thoughts…

What do you think? Would you use this approach? What do you like / dislike about this approach?

Before You Go…

Answer the following with a yes, no or maybe…

1. I have learned something about validation during this session.

2. I will change some of the assessment processes I currently use.

3. I am more confident about validating with others and sharing my assessment processes.

Is there anything else you need to know about validation?

Thank you and watch out for Foxes and Vixens!

Renae Guthridge – WA Training Institute

[email protected]

All pictures in these slides were taken from http://search.creativecommons.org/

NEXT SESSION – Session 5

MR1 MR2 MR3 MR6 MR7 MR8

The revised

regulatory standards

Train like a champion PowerPointless Taming the vocational

currency beast

Creating opportunities

for VETiS students in

rural areas

Industry engagement

Claire Werner Greg Mitchell Jane Goodfellow

Jason Washington-

King

Jodie Fitzsimmons David Hicks

The Meyvn Group

Major Mitchell

Productions

West Coast Institute of

Training

Centre for Training

Excellence

Department of

Education

Engineering and

Automotive Training

Council (EATC)

Tuesday 26th 1:00pm – 2:00pm