Competency assessment boot camp basics with tp

1

What is your biggest challenge in training &

competency assessment?

2

Competency Assessment Boot Camp Basics for Labs

Caer Vitek

Education Coordinator for Graduate Medical Education

Department of Laboratory Medicine and Pathology

May 5, 2009

3

Disclosures

• Caer Vitek has nothing to disclose.

4

I have previously downloaded or reviewed today’s session.

a) b)

11a) True

b) False

5

How long have you worked in a laboratory?

a) b) c) d)

0 000a) Under two years

b) 2-4 years

c) 5-10 years

d) More than 10 years

6

My involvement in training laboratory staff

a) b) c) d) e)

0 0 000a) Totally responsible for developing and providing training

b) Key developer of training

c) Work as part of a training team

d) Do some training as part of my job

e) I am not involved in training

7

Pre-test

8

Analyzing the results of a proficiency test is an example of a(n) ___________ objective.

a) b) c) d) e)

0 0 000a) Cognitive

b) Formative

c) Psychomotor

d) Affective

e) Confirmative

9

Multiple choice tests are

a) b)

11a) the most effective form of assessment.

b) only to be used to assess objective factual knowledge.

10

Learning Objectives (At the completion of today’s session, the audience members will be able to:)

• Define key characteristics of effective learning objectives

• Revise learning objectives to enhance assessment

• Select assessment tools based on learning objectives and level of skill/knowledge

11

Why are objectives important? (WIIFM)

• Clarify expectations for– learners – trainers

• Provide a framework to evaluate the learning

12

How do you create training for labs?

• Identify need

• Develop plan

• Gather materials

• Assessment

• What are new task(s) or process(es)?

• How do my learner(s) learn best?

• What will I need to develop in order to master this new task?

• How will I know if they’ve learned it?

13

Fundamental training for labs?

• Identify need

• Develop plan

• Gather materials

• Assessment

• Evaluate

Example:• New process

• Map out process, identify critical steps

• Bones, treats, leash

• Competency

• Successes & revisions

14

Basic #1: Be clear

• Be clear:– Do I know my learner?– Are the goals clear?– Are the objectives clear?– Is the plan clear?

15

Goal, objective or strategy?GOAL=

broad, generalized statement about what is to be learned

OBJECTIVE=

Specifies what the learner is to accomplish

16

What is the strategy?GOAL=

broad, generalized statement about what is to be learned

OBJECTIVE=

Specifies what the learner is to accomplish

STRATEGY=

Specifies how the learner will learn

17

View DVDs of presentations on . . . .

a) b) c) d)

0 000

a) Goal

b) Objective

c) Strategy

d) Assessment

18

Basic #2: Know your goal

• Then consider strategy & assessment

19

Authentic assessment

What do you use?

20

How will I assess my learners?

• Common mistake:

• Choose the assessment tool, then force-fit the objectives

• Better: Work backwards

OR

concurrently

21

Basic #3: Choose from a toolbox

Assessment Toolbox• Observe skills & behaviors• Rating scale• Checklist• Direct testing• Case studies• Unknowns• Others: Rubric, Portfolio

22

Know when they know(assessment of competency)

• Match the learning/behavior with the assessment

• Target behavior performance reflects how well the trainee can carry out a task(s)

• Role of standards

• Importance of self-assessment

23

NEW Version

http://www.odu.edu/educ/roverbau/Bloom/blooms_taxonomy.htm

Bloom’s Taxonomy


24

Writing learning objectives• A

• B

• C

• D

25

Writing learning objectives• Audience= learner

• B

• C

• D

26

Audience

• Who is your learner?

27


• Behavior= able to know or to do

• C

• D

28



• Condition= circumstances under which objectives must be completed

• Degree= standard of acceptable performance

29



• Condition= circumstances under which objectives must be completed

• Degree= standard of acceptable performance

30

Types of objectives

Cognitive(Knowledge)

Psychomotor(Skill)

Affective(Emotion)

31

NEW Version


Bloom’s Taxonomy


32

Bloom’s Taxonomy for Cognitive Objectives

Remember: Can the learner recall or remember the information?

define, duplicate, list, memorize, recall, repeat, reproduce state

Understand: Can the learner explain ideas or concepts?

classify, describe, discuss, explain, identify, locate, recognize*, report, select, translate, paraphrase

Apply: Can the learner use the information in a new way?

choose, demonstrate, dramatize, employ, illustrate, interpret, operate, schedule, sketch, solve, use, write

33

Bloom’s Taxonomy for Cognitive Objectives (cont.)

Analyse: Can the learner distinguish between the different parts?

appraise, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test

Evaluate: Can the learner justify a stand or decision?

appraise, argue, defend, judge, select, support, value, evaluate

Can the learner create new product or point of view?

assemble, construct, create, design, develop, formulate, write

34

Behavior

35

And now, a word about verbs:

• Understand• Appreciate• Learn• Improve• Increase• Motivate• Know• Really know

• Approach • Grasp the

significance of• Be familiar with• Expand horizons• Become• Grow• Think critically• Recognize*

36

And now, a word about verbs:

• Understand• Appreciate• Learn• Improve• Increase• Know• Really know

• Use judiciously for goals!

• Approach • Grasp the

significance of• Expand horizons• Become• Grow• Think critically• Recognize*

37

Conditions

• Learning – Environment – Materials– Standard Operating Procedures (SOP)

38

Degree=Standards

How • much?• many?• often? • well?• will I know when there is

success?

• Examples:• Until the syringe is empty.• All 96 wells• At the start of every run.• No more than 1 error.• When the tubes are in

position and the red light is on.

39

Templates

• After the training period the learner will be able to perform ________(task)_________. The task must be performed under the following conditions: ____(condition)______, _____(condition)_____, and ____(condition)______. The following standards must be met: ____(standard)_____, _____(standard)____, and ____(standard)_____.

• Given a ____(condition)_____, ____(condition)____, and ____(condition)_____, perform _______________(task)_________________. The task must be performed as ____________(standards)_____________.

40

Knowledge-based objective

Compare and contrast the different methods used for glucose, electrolyte, and coagulation (ACT, PT, PTT) testing in the critically ill patient, including differences between whole blood and plasma-based testing

41

Rewriting Learning Objectives to be assessable

Before: The learner will appreciate the principles of quality assurance.

After:

42

At the completion of this training, the learner will be able to:

1. Demonstrate/describe a systematic approach to reading PT evaluation reports.

2. Identify two resources available to assist with report interpretation.

3. Demonstrate/describe a systematic method for investigating PT failures.

4. Identify three types of unacceptable PT results that warrant investigation.

5. Differentiate appropriate from inappropriate methods of investigating unacceptable PT results.

6. Accurately complete a Proficiency Test Failure Reporting form.

What recommendations would you offer to enhance these?

43

The learner will list the safety features on the FUTURO 5000.

a) b) c) d) e) f)

0 0 0000

a) Remember

b) Understand

c) Apply

d) Analyze

e) Evaluate

f)f) CreateCreate

44

The learner will compare the safety features on the FUTURO 5000 to the FUTURO 2000.

a) b) c) d) e) f)

0 0 0000a) Remember

b) Understand

c) Apply

d) Analyse

e) Evaluate

f)f) CreateCreate

45

Basic #4: Know when they know

How well must they master the information or technique?

• often?• well?• many?• much?• will we know it is OK?• (combination)

46

Know when they know(assessment of competency)

• Knowledge & Comprehension:– Quizzes and exams

• multiple choice, matching, fill-in-the-blank, • direct observation

• Application, Analysis & Evaluation:– Real-life, authentic problems

• proficiency testing, unknowns

• Creation:– Portfolio, presentations

47

Keys to success

• Build into training: – Steps for the learner to self-assess

48

Assessing lower-level competencies

CONTENT RECALL APPLICATION

Fact

Concept

Principles/Rules

Procedure

Interpersonal

AttitudePerformance-content matrix

49

Assessing higher-level competencies

• What does an authentic problem look like in your lab?

• What does a learner need to have/do to solve it?

50

51

Given a pipette____________ 100 ul accurately.

a) b) c) d)

0 000a) Measure and aliquot

b) Dispense

c) Dispell

d) Mix a suspension

52

The learner will list and describe the safety features on the FUTURO 5000.

a) Remember

b) Understand

c) Apply

d) Analyse

e) Evaluate

f)f) CreateCreate

Beware of compounding-don’t do it

53

What are the effects of compounding?

• Compounding: using 2 or more behavioral and/or knowledge verbs in one objective– “list” and “compare”– “compare” and “contrast”

• Makes assessment difficult, challenging or impossible!

54

Basics

1. Be clear

2. Know your goal(s)

3. Use a good toolbox

4. Know when you’ve achieved the goal (and when you haven’t)

55

RESOURCES

• Revised Bloom’s Taxonomy: http://www.odu.edu/educ/roverbau/Bloom/blooms_taxonomy.htm



56

Contact information:

Caer Vitek: [email protected]

mailto:[email protected]

57

Performance-based objective

Analyze, contrast and evaluate various service options that meet clinical needs for rapid turn-around time and analytical precision,

Review advantages and disadvantages of each option, present your conclusion.

58

Characteristics of learning objectives

• Specific

• Measurable

• Acceptable (to you and your learner)

• Realistic

• Time-bound

59

Rules for Writing Single Response Questions

1) Focus on an important topic, usually a common orcritical clinical problem; avoid esoterica and “zebras”

2) Assess application of knowledge, not recall

3) Pose clinical decision-making tasks that are within theeducation/experience of examinees

4) Pose a clear question in the lead-in– can you answer it without looking at the options?

5) Use homogeneous distractors

6) Avoid technical flaws

60

Grammatical Cues

The option(s) does not flow from the stem

The minor differences among organisms of the

same kind are known as

A. Heredity

B. Variations

C. Adaptation

D. Natural selection

61

Logical Cues

A subset of options are collectively exhaustive.

Crime is

A. Equally distributed among the social classes

B. Overrepresented among the poor

C. Overrepresented among the middle class and rich

D. Primarily an indication of psychosexual maladjustment

E. Reaching a plateau of tolerability for the nation

62

Absolute Terms

Terms such as ‘always’ or ‘never’ are used in options.

In patients with advanced dementia, Alzheimer’s type, the memory defect

A. Can be treated adequately with lecithin

B. Could be a sequela of early parkinsonism

C. Is never seen in patients with neurofibrillary tangles

C. Is never severe

D. Possibly involves the cholinergic system

63

Long Correct Answer

The correct answer is longer, more specific, or more

complete than the other options.

Secondary gain is

A. Synonymous with malingering

B. A frequent problem in obsessive-compulsive disorder

C. A complication of a variety of illnesses and tends to prolong many of them

D. Never seen in organic brain damage

64

Word Repeats

A word or phrase is included in the stem and correct

answer.

A 58-year-old man with a history of heavy alcohol

use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called

A. Depersonalization

B. Derailment

C. Derealization*

D. Focal memory defect

65

Options are long, complicated or doubled

Systematic geography differs from regional geography in that

A. Systematic geography deals, in the main, with physical geography, while regional geography concerns

itself essentially with the field of human geography

B. Systematic geography studies a region systematically while regional geography is concerned only with descriptive account of a region

C. Systematic geography studies a single phenomenon in its distribution over the earth in order to supply generalizations for regional geography, which studies the arrangement of phenomena in one given area*

66

Numeric data are not stated consistently

Following a second episode of infection, what is the the

likelihood that a woman is infertile?

A. Less than 20%

B. 20% to 30%

C. Greater than 50%

D. 90%

E. 75%

67

Options in an nonlogical order

The population of Denmark is

A. 2 million

B. 15 million

C. 4 million

D. 7 million

68

“None of the Above” is used as an option

Which city is closest to New York City?

A. Boston

B. Chicago

C. Dallas

D. Los Angeles

E. None of the above