Assessment module - Wikispacesof... · Educational measurement assessment units Aschroft and Foreman- Peck (1994) point out that assessment is a key element in the effective management

Principles of assessment

Assessment

module

Norbert Boruett

A Pragmatic approach

With Norbert Boruett

Educational measurement assessment units

Pre- amble

Everything that be counted does not necessary

count: everything that counts cannot necessarily

be counted. Albert Einstein, physicist 1879-1955

One test does not improve learning anymore than

a thermometer cures fever. we should be using this

tests to get schools to teach more of what we want

students to learn, not as a way to punish them.

Hubert, professor of education, Columbia

University


Assessment- try to visualise

why should we assess?

what should we assess ?

How should we assess and

when should we assess?


Effects of assessment on learning

and teaching Assessment influences what is taught and

what is learned. Often teaching is artificially

divide from assessment. The two teaching

and learning are linked by

feedback(Harden,1992).

. We shall discuss more on feedback on

subsequent parts of this module


Effects of assessment

Torrance (1993) asserts that changing theories and methods of assessment have been the focus of significant attention for some years now. Curriculum developers have realised that real change will not take place in schools if traditional paper and pencil tests, be they essay or multiple choice, remain unchanged to exert a constraining influence on how teachers and pupils approach curricula.


Aschroft and Foreman- Peck (1994) point

out that assessment is a key element in the

effective management of learning than any

component of the learning process.

Curriculum developers have noted that

positive change will not take place in

learning institutions if assessment does not

change (Tornance,1993).


and teaching- cont


Miller (1976) puts it more succinctly by asserting

that to change the curricular without changing

examination will achieve nothing.

He further observes that changing the

examination systems without changing the

curriculum had a more profound impact upon the

nature of learning than changing curriculum

without altering the examination.


and teaching- cont



and teaching- cont The bicycle is a useful model when

considering assessment and teaching. Teaching and learning are represented by front wheel, assessment by the rear wheel. Disasters may occur when the two wheels go in different directions. If the rear wheel is omitted altogether only one wheel remains: difficulties and problems arise. In the same way problems may arise if a curriculum is designed with no assessment component


Assessment and evaluation-?.

Aschrof and Foreman – Peck (2000)

highlight that assessment is often confused

with evaluation. Assessment can be

conceptualised to mean judgment about the

progress (formative assessment) or

achievement (summative assessment) of a

particular course of study or a particular

teaching intervention (Nayar, 1995).


Formative assessment

There is a problem with formative

assessment since continuous assessment is

frequently used summatively.

Breaks learning into manageable units

Allows repeated attempts to master the

content of each module

Is not perceived as threatening (low stakes)


Summative assessment

Is the end point examination

Can block intended career progression(high

stakes)

Is perceived as threatening


Criterion and normative

assessment Assessment can be seen as criterion or

normative based. Brown (1994) postulates that criterion based assessment tells us what has been achieved. He further articulates that this type of assessment should have a description telling us what has not been achieved and the reason behind the failure. Put, succinctly criterion referencing is designed to check what a student has mastered in an area of knowledge or skill.


Tools in assessment and

evaluation Brown (1995) asserts that the same tools

may be used to assess and evaluate at the

same time, for example it is possible to use

one tool to evaluate a teaching method and

assess a student. If a student does well in an

examination, it follows then that the

teaching method was appropriate. This

example implies the function as an

evaluation of the teaching received.


Conclusion

As a principal objective of medical

education is to produce a competent

physician , then , unquestionably , the

raison d'être for the evaluation process is to

assess the competency and not the rank

order of students. To this end, criterion –

reference testing is necessary and must

become the principal method of evaluation

within medical education( Turnball,1989)


Who designs an administers the

assessment ? Traditionally the teacher has been

instrumental in executing this task.

Perhaps you may now be aware that currently there exist assessment methods where students take sole responsibility and much-pronounced role in their assessment

Examples here include portfolio-learning, self-assessment, peer assessment (Brown, 1992; Boud, 1995).


Assessing ability- competence

Hubbard adage states that:

There is something more scarce,

Something finer far,

Something rarer than ability.

It is the ability to recognise ability


OSCE-as an answer

Lumley (etal., 1999) assert that OSCE

brings a new dimension to assessment of

medical training. Of particular importance

is the ability to examine practical and other

skills in a unified, measurable and

reproductive fashion.


OSCE CONT

They are effective in assessing knowledge

and practical skills and ensure that each

student is presented with the same material,

thus providing a uniform evaluation with

the consistent marking of all those involved.

Do you recall the Kenyan experience in

OSCE?


Why OSCE’S?

Construct validity is the ability of the

OSCE to differentiate students ability , or

follow a student’s progress before and after

a course of instruction.

Content (criterion- based) validity assesses

the value of the station in reaching its

specified objective.


Why should students be

assessed?

Safeguarding the public

Feedback to the students

Certification

Monitoring the programme


Feedback

Rowntree has laid down a claim that feedback,

or knowledge of results, is the lifeblood of

learning.

Feedback is valuable – hence it should not consist

of notification of pass or failure. Feedback

should be given immediately and comprehensive

at the end of an examination (Dunnington et al.,

1992). Feedback should be given positively

(Amiel and Lunenfeld, 1992).


Guidelines for giving feedback

Timing of feedback

· . Methods of providing feedback

Students checked their answers against the correct ones written on the board.

Students discussed the questions and answers with an instructor.

Students were assigned to textbook materials relating to questions they had answered incorrectly


Feedback methods cont

Give student’s checklist with examiners

ratings

Watching video tape of own performance

Discussion with examiner

Watching video tape of expert

Receiving feedback from patient or

standardised patient

Reading computer generated report of

performance


What should we assess?

Knowledge (Cognitive)

Skills (Psychomotor)

Attitudes (Affective


How should we assess?

Validity- ensure the examination is valid

Reliability- ensure that the exam is reliable

Feasibility- set a realistic exam that can measure what you want to.


When should students be

assessed?

During the course of training

At the end of the course


Formative assessment

The scoring of formative assessment

should be strictly criterion referenced so as

to indicate to students their level of

performance. Aschrof and Foreman – Peck

(2000) asserts that formative assessment

should support student learning , records of

achievement , learning conflicts, portfolio

record of competence.


quip

Examinations are formidable even to the

best prepared , for the even the greatest fool

may ask more than the wisest man can

answer.

Charles Caleb Colton ,1780-1832


Concepts associated with assessment

Validity- measuring what you purport

(want)to measure

Reliability- consistency.


Types of validity

Face validity-

Content validity

Predictive validity

Consequential validity

Construct validity


Type of validity Test facet being

measured

Questions asked

Face validity Compatibility with

educational philosophy

What is the test face

value? Does it match up

with educational

intentions?

Content validity The content of curriculum

Does the test include a

representative sample of

the subject matter?

Construct validity The ability to differentiate

between groups with

known difference in

ability(beginners versus

experts

Does the differentiate at

the level of ability

expected of candidates at

the stage in training?


Type of validity Test facet being

measured

Questions asked

Predictive validity The ability to predict an

outcome in the future eg

professional success after

graduation

Does the test predict

future performance and

level of competence?

Consequential validity The educational

consequence of the test

Does the test produce the

desired educational

outcome?


Reliability

Inter-rater reliability- which correlates the

consistency of rating of performance across

different examiners

Inter-case reliability which quantifies the

consistency of performance of candidate

across the cases


Reliability

Parallel form reliability

Test retest reliability


Estimating reliability

Split half

Cronbach alpha coefficient

how can we handle reliability issues?


Norm referencing and criterion

what is the difference ?


Why use norm referencing tests?

As you answer that - think of the inherent

dangers of norm referencing

Outline some of them


Criterion-referenced tests

How is this different from the former? And

what are the advantages and disadvantages

of this referencing visa –vis the former


What are the practical issues od

assessment design Utility=

reliabilityXvalidityXfeasbilityXacceptabilit

yXeducational impact- Cees van der

Vleuten 1996


References

1. Talpot M. Monkey see and monkey do: a

critique of the competency model in

graduate medical education. Med Edu

2004:38 587-92

2. Wass V,Vleuten CPM van der,Shatzer

J,Jones R. Assessment of clinical

competence.Lancet 2001: 357 945-9


Modernisng medical cereers. www.mmc.uk

Southgate L,Grant J. Principles for

assessment system for postgraduate

training.Post medical training Board.

www.pmetb.org.uk

Miller GE. The assessment of clinical

skills/comptence/performance. Academic

med 65 s 63-7

http://www.mmc.uk/

http://www.pmetb.org.uk/

Documents

Assessment module - Wikispacesof... · Educational measurement assessment units Aschroft and Foreman- Peck (1994) point out that assessment is a key element in the effective management