Upload
devan-pannen
View
199
Download
3
Embed Size (px)
Citation preview
Evaluation
Principles and
Objectives
MEU WORKSHOP
Evaluation Principles and Objectives
Objectives
Teachinglearning methods Evaluation
Relevance
• Why
• What
• When
• Whom
• How
Is evaluation necessary?
• To make sure that the student will be able to do the job competently
• Exams encourage the students to work harder
• To guide the teacher and student about which part of the course has been successful and which part needs to be improved
Student evaluation
• Measurement: application of mathematical tools for finding the degree of achievement
• Example: awarding marks for MCQ
Assessment
• This term is used for those attributes which do not lend themselves to precise measurement and where some subjective decisions are involved
• Example: marking of essay type questions
Evaluation
• Involves passing a value judgement based on the information obtained from measurement and assessment
• It is a major component of the curriculum
Functions of evaluation
• Feedback for improved learning
• Predictive
• Selection
• Grading
• Program evaluation
Types of evaluation
• Formative: helps the learner and the teacher to know the progress of the student
• Summative: end of term evaluation
What is the role of internal assessment?
Abilities to be evaluated
• Evaluate all three domains of learning
• Weightage allotted to each component will vary depending on the subject area under consideration
• Evaluation is not the end – rather it is the means to further the effectiveness of an educational programme
• It has to be planned keeping the educational objective in mind
Evaluation tools
• Should be appropriate for the learning outcomes to be evaluated
• The characteristics of an evaluation tool are:• Validity
• Reliability
• Feasibility
• Relevance
• objectivity
Validity
• A tool is valid if it measures what it is supposed to measure
• Example: to determine whether a student can prepare a thin blood smear.
Types of validity
• Content validity
• Criterion validity and
• Construct validity
• Content validity:
• indicates synchronization between content of the test and content of teaching
• To ensure that content validity is measured,
– prepare a list of content matter to be tested
– Assign weightage for each
– Prepare table of specification
– Create test for each
How to build-in content validity
Prepare a table of specifications
Topic Weightage
A 15%
B 10%
C 5%
D 20%
E 30%
F 20%
Total 100%
Table of specifications
• Items required 01
• Use of gloves 01
• Size of the drop of blood 01
• Location of blood drop on the slide 01
• Holding the spreader 01
• Way the smear is spread on the slide 02
• Quality of the smear 02
• How the smear is dried 01
• Max marks 10
Factors influencing validity
• Unclear directions
• Ambiguous phrasing of questions
• Inappropriate level of difficulty
• Inappropriate question for the outcome to be tested
• Too short a test
• Insufficient time
Reliability
• Consistency of measurement
• The degree of reproducibility determines the reliability of an evaluation tool
– Test-retest reliability
– Equivalent-forms
– Split halves reliability
– Marker reliability
Measures to improve reliability
• Optimum time of test
• Use appropriate levels of difficulty
• Maintain conditions of test constant
• Ensure objectivity of scoring
• Ensure validity of the instrument
• A reliable test need not necessarily be valid!
Objectivity
• Structuring of questions
• Preparing model answers
• Agreeing on the marking scheme
• Having papers independently valued by two examiners
Feasibility
• Time and resources required
Relevance
• Appropriateness of the process of evaluation with reference to the tasks to be performed by the students after qualification
• Reflect the health needs of the society
How do you choose an appropriate evaluation tool?
• Purpose
• Domain to be tested
• Number of students
• Time allotted
• Resources
• Ease of administering the test
• Ensuring validity and objectivity
Steps of evaluation
Define objectives
Provide teaching-learning experience
Select measuring instrument
Decide marking
Administer test
score test
Analyse result
Take decision
Domain-wise matrix of evaluation methods
• Cognitive domain:
– Written and oral
• Essays- Modified essays
• Short answers and very short answer questions
– Objective items
– Simulated patient management problems
• Psychomotor domain:
• Observation
• Practical Clinical
• OSPE / OSCE
• Affective Domain:
• Observation
• Rating scales
• Checklist
• Questionnaires
• Logbook
• Group discussion
What makes a good evaluation?
• Does the evaluation comply with the regulations of
the course
• Does it test important skills and abilities ( is it valid)
• Are you sure that the marks gained by each student
is accurate ( is the marking reliable)
• Does the evaluation give you information that
will help the students to learn better and help
you to improve your teaching
• Is it reasonably economical in terms of
material and time
EvaluationEssay and SAQ
Evaluation of knowledge
• As a general rule, early in the course, factual
knowledge is presented to the student and as the
course develops higher domains of learning come
into picture
Current system of examination
• Evaluation of knowledge: Theory paper:
– Two long essays 2 X 10= 20 marks
– Six short notes: 8 X 5 = 40 marks
– Short answer questions: 10 x 2 = 20 marks
Evaluation of Cognitive skills
• Commonest tool is the traditional essay question
– Unstructured essay question
• Student has the freedom within the subject context, to
determine the nature and scope of the answer
• Example: write an essay on protein energy malnutrition
• Free response questions
• Subjectivity in marking is high, hence unreliable
• Advantages:
– Evaluates higher domains of learning and comprehension
of the student
– Relatively easy to frame
– Allows students free and effective expression
– Ability to organise ideas
• Disadvantages
– Low validity and reliability
– Limited range of application
– Lack of objectivity
– Inconsistent and time consuming scoring pattern
– Hardly any feedback to the teacher
– Scope for guess work by the students
– First answer and its assessment influences the subsequent ones
Structured essay questions
• Restricted response questions
• Example: write an essay on PEM covering the causes, classification, clinical diagnosis and initial management in the hospital
Modified essay question
• Problem solving question
• Short history is given based on which questions are asked
• Requires the student to apply what he has learnt, in the context of a given situation
Problem solving question
• A 4 year old boy presents with H/O epistaxis on and off since 3 years. On examination his knee joints are swollen. Mother gives H/O similar complaints in the older sibling
• What are the possible causes for this type of presentation?
• What investigations will you order to arrive at a diagnosis?
• What advise will you give to the mother?
Guidelines to write better essay questions
• Match the question to the specific learning outcome.
• The objective of learning and objective of testing
should be the same
• The more precise and clear we are with our learning
objectives, the easier and better it will become to
formulate good essay questions
• The question should clearly specify to the student
what is expected of him
• Phrase the question in a simple, clear language
• Indicate the weightage to different parts of the
question. This helps the student to effectively plan
his answer
• Prepare a checklist and allot marks to different points
to be included in the answer
• Mark essays question by question rather than
student by student, to compare the performance of
each student on that particular question and not as a
whole
• Mark the essay question by more than one examiner
and take average marks
• Grading instead of marking will further minimize the
variation between examiners
• All students should answer the question since
construction of optimal questions of equal difficulty
is not possible. Hence comparison of scores of two
students becomes invalid
• Avoid distractions during scoring
Short answer questions
• Tests lower domains of learning
• Student may be required to write a word, phrase, a
sentence to complete a statement
• Longer the expected answer, more is the subjectivity
involved
Types of SAQ
• Completion type
• Open SAQ
Merits of SAQs
• They are easy to construct when compared to MCQs
• They are more specific than essays and hence more
reliable
• They are quicker to answer. The students can be
tested on a wide range of topics in a short period
• They can be marked quickly
• SAQs are less prone to guessing than MCQs
• It is possible to construct a checklist for responses to
ensure intermarker reliability and thus objectivity
• Time taken to answer SAQ is more than for MCQ
• Number of SAQ in a given time is reduced compared
to MCQ
• Reliability of a test is directly proportional to the
number of questions
• Subjectivity in open SAQ
Group Activity I
• Each group is given two essay questions and two short answer questions
• Discuss within your group and write your comments
• Time allotted 10 minutes
• Presentation: 4 X 6 minutes: 25 minutes
• Total time: 35 minutes
Group Activity II
• Prepare two structured essay questions
• Two short answer questions
• Time allotted 10 minutes
• Presentation by each group: 4 X 5 = 20 min
• Total time: 30 minutes
Group activity I – Group A
• Essay questions:
• 1. Write in detail about oedema
• 2. Give a comprehensive account about the thoracic part of the oesophagus
• SAQ:
• 1. DNA
• 2. Cholecystitis
Group activity I – Group B
• Essay questions:
• 1. Write what you know about family planning
• 2. Discuss the role of endocrines in the regulation of human growth
• SAQ:
• 1. Right atrium
• 2. List a few haemoparasites. Can you draw them
Group activity I – Group C
• Essay questions:
• 1. Write an essay on SLE
• 2. Amyloidosis
• SAQ:
• 1. Rifampicin
• 2. PCV tubes
Group activity I – Group D
• Essay questions:
• 1. Write an essay on Senile cataract
• 2. Inguinal hernia
• SAQ:
• 1. POEMS
• 2. R S cell