TF – SP Capability Development Programme for Competency-Based Assessment

TF – SP Capability Development Programme for Competency-Based Assessment

Facilitator: Dennis Sale

Programme Objectives

• Identify the main components of a competency-based qualification system (e.g., Singapore Workforce Skills)

• Explain the process and principles of good assessment practice

• Analyse the standards for developing and conducting competency-based assessment

• Produce an assessment plan for a element of competence

• Produce assessment methods and tools for an element of competence

• Validate an assessment plan and accompanying assessment tools

• Produce a portfolio of evidence for meeting the competency standards for assessment

• Identify the importance of verification in the quality assurance of assessment practices

• Identify the role and duties of an Internal Verifier

Competency Based Approach

Key features:

• Meeting specific work related functions

• Meeting a defined standard (at different levels) of competence

• Assessment is mainly through direct performance in real work activity

Singapore Workforce Skills Qualifications (WSQ) - Competency-Based System

Competency Unit

CompetencyElement 1

CompetencyElement 2

CompetencyElement 3

Underpinning KnowledgePerformance CriteriaRange and ContextEvidence Sources



Competency Unit 1: Develop A Competency-Based Assessment

Competency Elements

1.Prepare an assessment plan

2.Develop competency-based assessment tools

3.Validate assessment plan

Competency Unit 2: Conduct Competency-Based Assessment

Competency Elements

1.Prepare for conduct of assessment

2.Prepare candidate for assessment

3.Carry out conduct of assessment

Conduct Competency-Based AssessmentCompetency Element Title1. Prepare for Conduct of Assessment

• How to prepare for conduct of assessment

Underpinning KnowledgeA competent individual needs to know and understand:

Performance CriteriaA competent individual must be able to successfully perform the following:

1.1 Confirm details of assessment plan with relevant stakeholders1.2 Ensure resources stipulated in the assessment plan are available, safe and in proper working conditions1.3 Ensure the set up of assessment site is in accordance with assessment plan and workplace health & safety guidelines1.4 Ensure the assessment tools required for assessment are available for use

Conduct Competency-Based AssessmentCompetency Element Title1. Prepare for Conduct of Assessment

Range & Context• Confirm details of assessment plan with relevant stakeholder (e.g., evidence requirements, resources needed for assessment, organisational requirements)• Ensure availability of resources ( e.g., tools equipment , technical support, etc) and assessment tools (assessment instruments, tools, etc)

Evidence SourcesExamples of evidence needed to meet the standards required for this element

ProcessObservation of work, which includes• confirming details of assessment plan with stakeholders• checking assessment site is appropriately set up and assessment instruments are available for use

Product• Assessment site is set up according to assessment plan

A Structured Assessment Process

Stage 1 Stage 2a Stage 2b Stage 3 Stage 4

Planning for assessment with candidate

Reviewing performance and knowledge evidence

Judging all evidence against agreed standards

Recording decisions and giving feedback

Contributing to quality assurance processes

Technical Principles of Assessment

Assessment must comply with the following Technical Principles of

assessment:

• Validity• Reliability• Flexibility• Fairness

These Technical Principles of Assessment must be fully addressed in the design and conduct of the assessment process as well as the

development of assessment items and instruments

Validity

In basic terms, validity refers to the ‘truth’ of the assessmentsmade. A valid assessment is one that measures accurately what itis we intend to measure.

For example, a valid driving test is one in which driving skills(Performance) is measured in typical traffic (Conditions) againstthe criteria established by the Motoring Authority (Standard)

Reliability

Reliability refers to the capability of a test to produce the same scores with different assessors (persons scoring the test)

Grade A

Grade A

Grade A

Assessor 1 Assessor 2

Assessor 3

Flexibility

Flexibility is concerned with the process of assessment, not the standard of the assessment.

For example, candidates can display the competence in a range of situations (e.g., on the job, simulated conditions) at mutually convenient times, etc), provided the competence is validly and sufficiently demonstrated.

Fairness

Fairness relates to a number of considerations in assessment. However, they are all concerned with ensuring that candidates, when being assessed, are provided with appropriate access to the assessment activities and are not unfairly discriminated against in the assessment process.

Unfair discrimination typically means discrimination based on criteria unrelated to the assessment activity itself, for example, gender or racialcharacteristics.

Fairness is a general concern throughout assessment, relating as much toproviding candidates with sufficient knowledge and time for assessment, to non-discriminatory processes in marking their work.

Rules of Evidence

Evidence of competence must be:

• Valid• Sufficient• Current • Authentic

Sufficient

Sufficient refers to ‘how much assessmentevidence’ do we need in order to feel confidentthat a candidate is competent in the area assessed?’

Current

Currency of evidence refers to how recent the evidence is generated and whether it fully relates to the most up-to-date knowledge, skills and practices for the work function being assessed.

This needs to be checked against the industry standardand any specific policy guidelines stated.

Authentic

Quite simply this refers to how sure we are that the work produced has been done by the candidate

In an examination, we can be more confident of authenticity

However, with assignments done by candidates in their own time, authenticity becomes a concern

Efficiency

Assessment can be time consuming. It is important to:

Identify assessment opportunities that enable the assessment of as wide a range of performance criteria as possible

Use methods that are efficient for what we are assessing

Avoid over-assessing – be clear about what is Sufficient evidence

Identifying relevant evidence

Assessors and candidates need to be clear on:• What types of evidence are needed to meet the specific

performance criteria

• What underpinning knowledge can be assessed either through performance or questioning (written or oral)

• Arrangements for the assessment (e.g., equipment, materials required)

Collecting Evidence of Competency 1

• Assessment of competence must involve a full demonstration of the competence – and should involve avariety of evidence types and contexts where possible

• At least one form of direct evidence should be used in judging practical performance (e.g., observation of simulated performance)

• Supplementary (indirect evidence) should be used in support of direct evidence to:– Extend range of evidence and possible performance in rare

situations (e.g., theft, breakdown) – Facilitate transfer to new situations/contexts– Assess underpinning knowledge

Collecting Evidence of Competency 2

• Evidence gathering methods should be culturally inclusive and take into account the language, literacy and numeracy skills of the participant

• Reasonable adjustments may be considered for participants with physical disabilities or special learning needs

• Evidence of competency must include occupational safety and health and other legislative aspects of the job

• Prior learning can be recognized providing it can be validated by accepted certification.

Assessing Underpinning Knowledge

• While to some extent underpinning knowledge is implicit in performance – it should be separately assessed to ensure understanding (the why behind the what and how)

• The use of verbal questioning (especially immediately after demonstrating competence) is an effectively way to assess underpinning knowledge

• Use oral or written work related questions in assessing underpinning knowledge (avoid MCQ’s)

Assessment Context

The assessment context is the participant’s current place of work and job role. A judgement of competence denotes that the participant is able to produce the desired outcomes reflected in the competency element within the requirements of the work role and environment.

Assessment may be undertaken on and off the job – the essential reference for the collection of evidence is the actual Competency Unit


Competency Element 1:

Prepare An Assessment Plan

Purposes of the Assessment Plan

• Show how the Unit is to be assessed - detailing actual assessment process, methods and instruments

• Calibrating assessment to actual competency elements and performance criteria, etc

• Provides explicit guide to trainers, trainees and other stakeholders as to how the assessment is to be carried out in relation to Principles & Rules of Assessment

Prepare An Assessment Plan

Meeting this competency requires you to:• Establish the purpose and context of assessment in consultation

with relevant people • Obtain information on the characteristics of candidates • Identify the types of evidence to be gathered based on the

competency standard• Determine what competences will be assessed (e.g., elements,

performance criteria, etc)• Select appropriate assessment methods and tools to collect the

required evidence• Identify assessment location, physical resources and specific

materials required for conduct of assessment



Develop Competency-Based Assessment Tools

Develop competency-based assessment toolsMeeting this competency requires you to:• Develop holistic and integrated assessment activities based on the

assessment plan• Develop specific assessment tools that are appropriately

contextualized for the target audience• Develop clear instructions for assessor and candidate on the

conduct of assessment• Develop assessment tools that meet the Principles of Assessment

and the Rules of Evidence



Validate Assessment Plan

Meeting this competency requires you to:• Identify the purpose, focus and approach to validation• Conduct the validation activities and address the critical

findings • Produce a report recommending improvements to the

assessment strategies, process and tools

Validate Assessment Plan



Prepare For Conduct ofAssessment

Meeting this competency requires you to:

• Confirm details of assessment plan with relevant stakeholders• Ensure resources stipulated in the assessment plan are available,

safe and in proper working conditions• Ensure the set up of assessment site is in accordance with

assessment plan and workplace health & safety guidelines• Ensure the assessment tools required for assessment are available

for use

Prepare For Conduct Of Assessment



Prepare Candidate forAssessment

Meeting this competency requires you to:• Explain the purpose and context of assessment to candidate

• Brief candidate on the assessment requirements and process, including clear instructions on the assessment to be taken

• Brief candidate on his rights and process of appeal to the assessment outcome

• Communicate in a clear and supportive manner to create a conducive assessment environment

• Confirm with candidates on any special needs he/she may have and explain how those needs will be addressed during the assessment

• Seek feedback from candidate and clarify his/her understanding of the assessment requirements

Prepare Candidate for Assessment



Carry Out Conduct of Assessment

Meeting this competency requires you to:• Use assessment plan and tools to carry out the conduct of assessment• Use evidence gathered to decide if the relevant criteria are met and make

assessment decision• Deal with limitations of evidence required, where necessary• Comply with the Code of Practice for Assessors when conducting assessment

and making assessment decisions• Record assessment outcomes promptly and accurately in accordance with

assessment system, policies and procedures• Provide clear and constructive feedback to candidate regarding assessment

decision and advice on how performance gaps can be overcome• Review assessment plan and process and provide feedback to appropriate

personnel for modification, if required

Carry Out Conduct of Assessment

When giving feedback to candidates

1. Let candidates give their opinion on the assessment first 2. Identify positives before areas for improvement3. Limit what you cover4. Be specific, not general 5. Focus on what can be changed/improved6. Give the candidate time to think and respond7. Keep to the standards8. Check for understanding9. Listen to how the feedback is received (including observation of non-

verbal communication)10. End on a positive (with encouragement)

Verification: Quality Assurance in Assessment

Verification is the main quality assurance process associated with assessment systems and practice - whether competence-based or otherwise

Quality Assurance is a vital part of any credible qualification system. It must ensure that everyone involved in the assessment of candidates is assessingcorrectly, working to agreed procedures and to an agreed standard of performance

Verification: The Range of Roles & their Relationships

External Verifier

Internal VerifierCoordinator

Internal Verifiers(programme level)

Assessors

Awarding Body

Centre

Centre

Centre

Workplace

Internal Verifier (IV): The key Role in Quality AssuranceThe internal verifier(IV) is the key person(s) for ensuring quality

of assessment practices at the institutional level (Centre)

The IV conducts internal quality assurance of the assessment

process by:

1. Carrying out and evaluating internal assessment and quality assurance systems

2. Supporting assessors

3. Monitoring the quality of assessors’ performance

4. Meeting external quality assurance requirements

Carrying out and evaluating internal assessment and quality assurance systems

This will involve:• Comparing own institution requirements with those of the external

awarding body• Identifying the outcomes needed by the agreed standards and their

consequences for internal auditing• Auditing existing administrative and recording arrangements -

modify/ change if necessary - to meet external audit requirements• Carrying out assessment standardization arrangements (e.g.,

sampling strategy)• Ensuring a procedure for complaints and appeals is in place (note:

consider this in local context)• Identifying problems, developing improvement plans

Sampling strategy

This will vary from institution, according to needs (e.g., size,

programmes offered, resources, etc) but it must be agreed with the

External Verifier.

The following need to be taken into account:• Size of sample to ensure reliability

• Assessment methods used by the centre (e.g., to ensure good principles of assessment are practice, etc)

• Assessors – number, experience, workload, location

• Candidate cohorts (e.g., full-time/part-time, different employees, etc)

• The programmes offered – particularly if new/revised

• All units must be included, but there may be a greater focus on any problem unit(s)

Standardizing assessment decisions

It is essential to ensure, as far as is possible, that:

• Assessors consistently make valid decisions• Assessors make the same decision on the same evidence base• All candidates are assessed fairly

This involves:• conducting moderation activities (e.g., a number of assessors

judging the same candidate’s evidence and comparing their perceptions and decisions)

Supporting assessors

This will involve:• Ensuring assessors have appropriate technical and

vocational experience• Ensuring that assessors are competent to assess• Identifying the developmental needs of assessors and

facilitating any necessary training• Monitoring progress and development of assessors

Monitor the quality of assessors’ performanceThis will involve:

• Ensuring that assessors are preparing and planning assessment effectively and efficiently (e.g., understand the standards and applying principles of good assessment, etc)

• Observing and questioning individual assessors to ensure good practice and consistency over time

• Checking records of the assessment process (from assessment planning to summative decisions)

• Giving assessors accurate and helpful feedback

Meet external quality assurance requirementsThis will involve:• Identifying how internal assessment will be checked externally and

ensuring these requirements can be met• Planning, collecting and analyzing information on internal

assessment decisions (note: this involves the secure keeping of all records relating to the assessment of candidates)

• Maintaining contact and meeting with the external verifier(s)• Providing feedback on the outcomes of meetings with the external

verifier to centre assessors• Acting on the feedback given by external verifiers

Checklist of the range of information needed for purposes of External VerificationIt is the Internal Verifiers responsibility to have the following information

quickly accessible:• Number of registered candidates per qualification level• Assessor details (e.g., CV’s, Assessor qualifications, development plans)• Candidate progress reviews and achievements• Assessment records and plans• Candidate support resources available (e.g., guide for candidate assessment)• Assessments sites used – where the assessments are carried out• Internal verification sampling strategy• Internal Verifier records, including feedback to Assessors and feedback from

Candidates• Record of claims for certification• Candidate assessment evidence (e.g., portfolios)

Dennis Sale can be contacted at:

Email: [email protected]

Telephone: (65) 6772 1490

Blog: dennissale.com

Documents

TF – SP Capability Development Programme for Competency-Based Assessment