The Competency The Competency Assessment Toolkit for Assessment Toolkit for

Professional PsychologyProfessional Psychology

Catherine L. Grus, Ph.D.,Catherine L. Grus, Ph.D.,Associate Executive Director forAssociate Executive Director for

Professional Education and TrainingProfessional Education and TrainingEducation Directorate Education Directorate

American Psychological AssociationAmerican Psychological Association

THANK YOUTHANK YOU

Many thanks to Many thanks to

• Nadya Fouad, University of Wisconsin Nadya Fouad, University of Wisconsin

• Nadine Kaslow, Emory University Nadine Kaslow, Emory University

• Benchmarks WorkgroupsBenchmarks Workgroups

• Assessment “Toolkit” WorkgroupAssessment “Toolkit” Workgroup

OverviewOverview

• Considerations in the assessment of Considerations in the assessment of competencecompetence

• Development of the ToolkitDevelopment of the Toolkit

• Putting the toolkit into practicePutting the toolkit into practice

Competence …Competence …

• “ … “ … the habitual and judicious use of the habitual and judicious use of communication, knowledge, communication, knowledge, technical skills, clinical reasoning, technical skills, clinical reasoning, emotions, values, and reflection in emotions, values, and reflection in daily practice for the benefit of the daily practice for the benefit of the individual and community being individual and community being served.”served.”

From Epstein and Hundert (2002)From Epstein and Hundert (2002)

Competence …Competence …

• … … is developmental depending on is developmental depending on stage of professional functioningstage of professional functioning

• … … context dependentcontext dependent

• … … executed according to ethical executed according to ethical principles, guidelines, standards, and principles, guidelines, standards, and values of the profession andvalues of the profession and

• … … requires public verificationrequires public verification

Competencies …Competencies …

– ObservableObservable– MeasurableMeasurable– ContainableContainable– PracticalPractical– Derived by expertsDerived by experts– And flexibleAnd flexible

are elements of competence that are are elements of competence that are ….….

Assessment of Competence Assessment of Competence - Considerations - Considerations

• Use of both “formative” and Use of both “formative” and “summative” evaluation methods“summative” evaluation methods

• Authentic evaluationsAuthentic evaluations• Real world tasks, meaningful Real world tasks, meaningful

activitiesactivities

• [High] Fidelity: Degree to which[High] Fidelity: Degree to which the assessment representsthe assessment represents

actual performanceactual performance

• Employ a range of perspectives Employ a range of perspectives (multi-informant) and methods(multi-informant) and methods

• Challenge of evaluating attitudes and Challenge of evaluating attitudes and valuesvalues

• Psychometric issuesPsychometric issues

• Feasibility/costs Feasibility/costs

• Lifelong self-assessmentLifelong self-assessment

What methods are What methods are commonly used to assess commonly used to assess students/ trainees?students/ trainees?

• In class examinations and course In class examinations and course “grades”“grades”

• Faculty/supervisor completes rating Faculty/supervisor completes rating scalescale

• Letters of recommendationLetters of recommendation

• EPPP, state jurisprudence examsEPPP, state jurisprudence exams

Developing Developing Faculty/Supervisor Faculty/Supervisor CompetenciesCompetencies• Training in “difficult conversations”Training in “difficult conversations”

– How to “critique constructively”How to “critique constructively”• Develop awareness of potential dual roles Develop awareness of potential dual roles

in evaluation and how that could influence in evaluation and how that could influence resultsresults– Benefits of having other faculty evaluate Benefits of having other faculty evaluate

students’ performancestudents’ performance– Explicit obligations to make evaluations Explicit obligations to make evaluations

accurateaccurate• Grade inflationGrade inflation• Letters of recommendation to “market” vs. evaluateLetters of recommendation to “market” vs. evaluate

• Knowledge of a range of assessment tools, Knowledge of a range of assessment tools, access to resources “toolkits”access to resources “toolkits”

Competency Assessment Competency Assessment for “Toolkit” for for “Toolkit” for Professional PsychologyProfessional Psychology

– Charge from APA Board of Educational Charge from APA Board of Educational Affairs: Develop a “Toolkit” for professional Affairs: Develop a “Toolkit” for professional psychologypsychology

– Purpose: Promote broader implementation of Purpose: Promote broader implementation of competence assessment and provide competence assessment and provide information about application of assessment information about application of assessment methods to the assessment of competencemethods to the assessment of competence

– Coordinated with Benchmarks Work GroupCoordinated with Benchmarks Work Group

Benchmarks Core Benchmarks Core CompetenciesCompetenciesFoundational Foundational

Competencies:Competencies:• ProfessionalismProfessionalism• Reflective practiceReflective practice• Scientific knowledge Scientific knowledge

and methodsand methods• RelationshipsRelationships• Individual and cultural Individual and cultural

diversitydiversity• Ethical and legal Ethical and legal

standards and policystandards and policy• Interdisciplinary Interdisciplinary

systems systems

Functional Functional Competencies:Competencies:

• AssessmentAssessment• InterventionIntervention• ConsultationConsultation• Research and Research and

evaluationevaluation• SupervisionSupervision• TeachingTeaching• AdministrationAdministration• AdvocacyAdvocacy

How are the Benchmarks How are the Benchmarks Organized?Organized?

• Core Foundational and Functional Core Foundational and Functional CompetenciesCompetencies

• Essential ComponentEssential Component: what are the : what are the critical elements of/what critical elements of/what knowledge/skills/attitudes that make knowledge/skills/attitudes that make up this competency? up this competency?

• Behavioral AnchorBehavioral Anchor: what would it : what would it look like if you saw it (essential look like if you saw it (essential component)?component)?

Components of ToolkitComponents of Toolkit

• Background and IntroductionBackground and Introduction• Assessment Method Fact SheetsAssessment Method Fact Sheets• Grid of Assessment Methods and Grid of Assessment Methods and

Competencies/Essential Components Competencies/Essential Components Best used forBest used for

Fact Sheet: Portfolio ReviewDescriptionA portfolio is a collection of products, gathered by the person being assessed …Application

Competency Applies to1 Predominant Use of2

Developmental Level3

Professionalism E F,S 2,3,4

Reflective Practice E F,S 1,2,3,4Implementation Deciding on form (e.g., web-based or hard copy)Psychometric PropertiesReliability has not been well established due to the variable content included in a portfolioStrengthsProvides a low cost assessment strategyChallengesRequires intense commitment of time and labor intensive for the assessors and the person being assessedFuture directionsCreate and implement training modules for assessors and people being assessed in the portfolio assessment strategy

Annual review

Case review

s

Process/outcome data

Rating F

orms

Consum

er surveys

OS

CE

Portfolio reviews

Ratings Perform

ance

Record review

s

Self-assessment

Simulations/role plays

Stand. patient interview

Stand. oral exams

Written exam

s

360 Evaluations

Intervention Knowledge interventions

2 3 3 2     1     2     1 1  

  Intervention planning

2 1 2 1 2 2   3 1   2 2 1    

  Skills 2 2 1 3 2 1 2 1 2   1 1 2      Implementation 1 1 1 1 1 1 2 1 2 1 2 2 2 3 1  Progress

evaluation3 1 2 2 2   3 2 2 1     2   2

Grid of Possible Assessment Grid of Possible Assessment Methods (Toolkit)Methods (Toolkit)

Assessment Methods in Assessment Methods in ToolkitToolkit• 360 evaluation360 evaluation• PortfolioPortfolio• OSCEOSCE• Structured Written & Structured Written &

Oral ExamsOral Exams• Case PresentationCase Presentation• Simulation/Role PlaySimulation/Role Play• Competence Competence

Evaluation Rating Evaluation Rating FormForm

• Self-AssessmentSelf-Assessment

• Ratings of live or Ratings of live or recorded performancerecorded performance

• Standardized Client Standardized Client InterviewInterview

• Client/Patient Client/Patient Process/Outcome Process/Outcome MeasureMeasure

• Consumer Satisfaction Consumer Satisfaction SurveySurvey

• End of Rotation End of Rotation Performance ReviewPerformance Review

Putting it all Together …Putting it all Together …

Benchmarks

Toolkit

Using the Toolkit and Using the Toolkit and Benchmarks to Guide the Benchmarks to Guide the

Assessment of Student Assessment of Student Learning OutcomesLearning Outcomes

Identify expected student learning outcomes using

competency model (e.g., Benchmarks)

Select assessmen

t method(s)

Expected level of competence

attainment and training plan

developed with trainee

Formative assessment

Student demonstrates problems with competencies

acquisition

Student demonstrates emerging

competencies

Training plan continues

Remediation plan developed

Summative evaluation to assess readiness for next level of

training

Identify expected student learning outcomes using

competency model (e.g., Benchmarks)

Program Program OutcomeOutcome: Students will demonstrate the : Students will demonstrate the following intervention (functional) competencies following intervention (functional) competencies prior to going on internship.prior to going on internship.

1.1. Knowledge of scientific, theoretical, empirical Knowledge of scientific, theoretical, empirical and contextual bases of intervention, and contextual bases of intervention, including theory, research, and practiceincluding theory, research, and practice

2.2. Formulates and conceptualizes cases and Formulates and conceptualizes cases and plan interventions utilizing at least one plan interventions utilizing at least one consistent theoretical orientationconsistent theoretical orientation

3.3. Clinical skills Clinical skills 4.4. Implements evidence-based interventions Implements evidence-based interventions

that take into account empirical support, that take into account empirical support, clinical judgment, and client diversity (e.g., clinical judgment, and client diversity (e.g., client characteristics, values, and context)client characteristics, values, and context)

5.5. Evaluate treatment progress and modify Evaluate treatment progress and modify treatment planning as indicated, utilizing treatment planning as indicated, utilizing established outcome measuresestablished outcome measures

Identify expected student learning outcomes using

competency model (e.g., Benchmarks)

Select assessmen

t method(s)

Annual review

Case review

s

Process/outcom

e data

Rating F

orms

Consum

er surveys

OS

CE

Portfolio review

s

Ratings P

erformance

Record review

s

Self-assessm

ent

Sim

ulations/role plays

Stand. patient interview

Stand. oral exam

s

Written exam

s

360 Evaluations

Intervention Knowledge interventions

2 3 3 2     1     2     1 1  

  Intervention planning

2 1 2 1 2 2   3 1   2 2 1    

  Skills 2 2 1 3 2 1 2 1 2   1 1 2    

  Implementation 1 1 1 1 1 1 2 1 2 1 2 2 2 3 1

  Progress evaluation

3 1 2 2 2   3 2 2 1     2   2

Grid of Possible Assessment Grid of Possible Assessment Methods (Toolkit)Methods (Toolkit)

Identify expected student learning outcomes using

competency model (e.g., Benchmarks)

Select assessmen

t method(s)

Expected level of competence

attainment and training plan

developed with trainee

Developmental LevelA. Knowledge of InterventionsReadiness for Practicum Readiness for Internship  

Essential Component: Basic knowledge of scientific, theoretical, and contextual bases of intervention and basic knowledge of the value of evidence-based practice and its role in scientific psychology

Behavioral anchor: Articulates the relationship of EBP to the science of psychologyIdentifies basic strengths and weaknesses of intervention approaches for different problems and populations

Essential Component: Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice

Behavioral Anchor:Demonstrates knowledge of interventions and explanations for their use based on EBPDemonstrates the ability to select interventions for different problems and populations related to the practice settingInvestigates existing literature related to problems and client issuesWrites a statement of one’s own theoretical perspective regarding intervention strategies

 

Identify expected student learning outcomes using

competency model (e.g., Benchmarks)

Select assessmen

t method(s)

Expected level of competence

attainment and training plan

developed with trainee

Formative assessment

Student demonstrates problems with competencies

acquisition

Student demonstrates emerging

competencies

Training plan continues

Remediation plan developed

Remediation Plan

Competency Domain/Essential Components

ProblemBehaviors

Expectations for Acceptable Performance

Trainee’s Responsibilities/Actions

Supervisors’/Faculty Responsibilities/Actions

Timeframe forAcceptablePerformance

AssessmentMethods

Dates ofEvaluation

Consequencesfor Unsuccessful Remediation

Identify expected student learning outcomes using

competency model (e.g., Benchmarks)

Select assessmen

t method(s)

Expected level of competence

attainment and training plan

developed with trainee

Formative assessment

Student demonstrates problems with competencies

acquisition

Student demonstrates emerging

competencies

Training plan continues

Remediation plan developed

Summative evaluation to assess readiness for next level of

training

For more information:For more information:

http://www.apa.org/ed/graduate/competency.htmlhttp://www.apa.org/ed/graduate/competency.html

Coming soon:Coming soon:

Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad, N. Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad, N. A., Hatcher, R. L., & Rodolfa, E. R. (in press) A., Hatcher, R. L., & Rodolfa, E. R. (in press) Competency Assessment Toolkit for Professional Competency Assessment Toolkit for Professional Psychology. Training and Education in Psychology. Training and Education in Professional Psychology.Professional Psychology.