Blueprinting and drafting examination questions, Liz Norman, ANZCVS Exam Writing Workshop, November...

Blueprinting and drafting

examination questions

Liz Norman

Massey University

Australian and New Zealand College of Veterinary Scientists,

Exam Writing Workshop, November 2013

Validity

Validity

The exam must elicit proper evidence of the

abilities of the candidate and we must evaluate the

evidence properly

Validity

• Not a property of a test, but relates to the

interpretations or actions made on the basis of

test scores

• Eg:

“The award of Fellowship of the College implies that the

awardee has sufficient knowledge and experience in a

particular area of veterinary science to entitle him/her to be

acknowledged as a specialist or consultant in that area"

Chain of inferences

1. Fellows have sufficient knowledge, skills, attitudes and judgement to be considered specialists

2. The things we can measure in examinations are things that are necessary to be a Fellow

3. A particular examination (all 4 components) is representative of all that we could measure in an examination

4. The examination score is a measure of achievement in the examination

5. The passing score of 70% correctly separates someone with sufficient knowledge, skills, attitude and judgement to be a Fellow from someone without

Specialist practice domain

All the

knowledge, skills, a

ttitudes and

judgements you

need to be a

specialist

Fellowship domain

All the

knowledge, skills, att

itudes and

judgements you

need to be a fellow

Content domain

Everything we could

possibly examine

Credentials domain

Everything that is

examined in the

credentials

document

Examination domains

What is examined in

a particular

examination in a

particular year

Why we need to blueprint

• Its important that we sample representatively

from the content domain (all that it is possible to

examine)

• This is so we can extrapolate performance to the

whole content domain

Content of questions

Content

• Scope of topics

• Other scopes eg species mix

Subject guidelines

• Represent the entire content domain (all that is

examinable).

• Specify both scope (breadth) and level (depth)

of the knowledge, skills, attitudes and

judgements required.

• For Fellowship some aspects are assessed in

the credentials documents.

Content

• Scope of topics

• Other scopes eg species mix

The candidate will have a detailed knowledge of:

The aetiology, pathogenesis and pathophysiology of

cardiac, renal, respiratory, alimentary, musculoskeletal,

endocrine, ophthalmological and neurological organ

dysfunction in the cat and the dog.

Level of questions

Level - depth

• Surface – deep

• Critical thinking

• Fact recall – applied

• Blooms taxonomy

• SOLO taxonomy

Critical thinkers

• seek reasons.

• attempt to be well-informed

• acknowledge credible sources

• consider alternative points of view

• withhold judgement until they have sufficient

evidence

• seek to be as precise as possible

Fact recall vs applied

Fact recall:Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts)

Applied (higher order)Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact

From: Peitzman et al. (1990). Academic Medicine, 65(9), S59-60.

create

evaluate

analyse

apply

understand

rememberRecall

Application

Blooms taxonomy

Blooms instructional verbs

• Create: compose, plan, propose, design, formulate, arrange, assemble, collect, construct, create, set-up, organise, manage, prepare.

• Evaluate: judge, appraise, evaluate, rate, compare, revise, assess, estimate

• Analyse: distinguish, analyse, differentiate, appraise, calculate, experiment, test, compare, contrast, criticise, diagram, inspect, debate, question, relate, solve, examine, categorise.

• Apply: interpret, apply, employ, use, demonstrate, dramatise, practice, illustrate, operate, schedule, sketch.

• Understand: translate, restate, discuss, describe, recognise, explain, express, identify, locate, report, review, tell

• Remember: define, repeat, record, list, recall, name, relate, underline.

SOLO taxonomy

http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/

Prestructural Question may be rephrased as the answer; almost completely

misses the point of the question.

Unistructural Able to identify, list, name, enumerate but does not describe,

explain, relate or elaborate multiple aspects of a response

Multistructural Able to list as well as describe distinct aspects of a response (such

as being able to describe aetiology, clinical features, management

of thrombotic stroke) but unable to explicitly explain causes for

observations; unable to present cause-effect relationships.

Relational Able to describe multiple aspects of a process and additionally

explain or elaborate observations into cause-effect relationships;

able to compare similarities and differences between apparently

distinct phenomena. This level is taken as suggesting that the

learner has understood.

Extended

abstract

Highly developed; able to explain mechanisms of phenomena and

apply this information to a novel context — able to develop novel

hypotheses, theories, and deduce principles; creative thinking.

Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149

Prestructural

Unistructural Paraphrase, define, identify, count, name, recite, follow simple

instructions, calculate, reproduce, arrange, recognise, find, note,

seek, sketch, pick

Multistructural Combine, classify, structure, describe, enumerate, list, do

algorithm, apply method, account for execute, formulate, solve,

conduct, prove, complete, illustrate, express, characterise

Relational Analyse, compare, contrast, integrate, relate, explain causes,

apply theory (to its own domain), argue, implement, plan,

summarize, construct, design, interpret (some senses), structure,

conclude, substantiate, exemplify, derive, adapt

Extended

abstract

Theorise, generalise, hypothesise, predict, judge, transfer theory

(to new domain), assess, evaluate, interpret (some senses),

critically reflect, predict, criticise, reason

Potter & Kustra (2012). http://www1.uwindsor.ca/ctl/system/files/PRIMER-on-Learning-Outcomes.pdf

Prestructural

Unistructural Able to identify, list, name, enumerate but does not describe,

explain, relate or elaborate multiple aspects of a response

Multistructural

Relational

Extended

abstract

surface

deep

Prestructural

Unistructural Able to identify, list, name, enumerate but does not describe,

explain, relate or elaborate multiple aspects of a response

Multistructural

Relational

Extended

abstract

recall

application

Prestructural

Unistructural Able to identify, list, name, enumerate but does not describe,

explain, relate or elaborate multiple aspects of a response

Multistructural

Relational

Extended

abstract

Quantitative change

Qualitative change

Blueprinting

Blueprinting

• Mapping to scope and level

Scope - breadth

PathophysiologyInvestigation and

diagnosis

Treatment and

management

Gastrointestinal P1Q1 P1Q1, P2Q4

Cardiovascular P1Q4 P2Q2 P2Q2

Nervous P1Q3, P2Q1

Endocrine P1Q3 P2Q3

Musculoskeletal P2Q5

Breadth across species

species number of Qs percent

small animal 15 52%

farm 5 17%

horse 4 14%

exotic 3 10%

lab 1 3%

all 1 3%

Level - depth

PathophysiologyInvestigation and

diagnosis

Treatment and

management

recallhigher

orderrecall

higher

orderrecall

higher

order

Gastrointestinal P1Q1P1Q1, P2

Q4

Cardiovascular P1Q4 P2Q2 P2Q2

NervousP1Q2,

P2Q1

Endocrine P1Q3 P2Q3

Musculoskeletal P2Q5

Question difficulty

Should questions to be difficult?

• Norm referenced vs standards based grading

Question difficulty

• Type of content

eg:

– Name the two most significant muscles involved for

mastication.

– Explain the physiological actions of insulin

– Explain the physiological actions of ghrelin

Question difficulty

• Type of content

• Level of task

– Complexity

– Abstractedness

Question difficulty

• Type of content

• Level of task

– Complexity

– Abstractedness

• Structure

Structure

1. Explain the aetiopathogenesis of hepatic lipidosis in cats. (12 marks)

2. The following drugs are commonly used in the management of hepatic lipidosis in cats. For each drug describe its mechanism of action AND any adverse effects or precautions for use.

a) Ursodeoxycholic acid. (3 marks)

b) S-adenosyl methionine (3 marks)

3. Outline the principles of nutritional management and fluid therapy for a cat with idiopathic hepatic lipidosis(12 marks)

Question difficulty

• Type of content

• Level of task

– Complexity

– Abstractedness

• Structure

• Resources

Question difficulty

• Type of content

• Level of task

– Complexity

– Abstractedness

• Structure

• Resources

• Wording

Written, oral, practical

Written Paper 1:

…designed to test the candidate’s knowledge of the principles of the subject as described in the learning outcomes….. Answers may cite specific examples where general principles apply, but should primarily address the theoretical basis underlying each example.

Written Paper 2:

…designed to (a) test the candidate’s ability to apply the principles of the subject to particular cases/problems or tasks, and to (b) test the candidates’ familiarity with current practices and current issues that arise from activities within the discipline…

Written, oral, practical

Practical examination

…designed to test practical aspects of the learning outcomes. …written answers …pertaining to case-based material presented. Candidates must demonstrate deep understanding and practical application of equipment used in the subject… Patient images or videos, radiographs, ultrasound images, CT studies, MR images, ECG recordings, EMG recordings, NCV recordings, cytology images as well as copies of pathology, radiography or other ancillary test reports are likely to be used during this examination.

Oral Examination:

…may cover case-based material or more general discussion, including areas of recent interest or controversy in the subject. …designed to test practical aspects of the learning outcomes. …be able to demonstrate the required level of knowledge …and to support their opinions with citations of the veterinary literature….. Images and pathology reports are likely to be used….

What’s different about oral Qs

• Thinking on your feet

• Can’t revisit or edit

• Tests verbal communication skills as well as

knowledge

• Interactive – can prompt and redirect, can ask

candidate to explain their answer

Question length

Question length

• More shorter questions?

Better content coverage but if not careful could have

less higher order Qs

Easier to mark