13
© Sanjaya Mishra, 2007 Evaluating a Programme in Distance Education Preparing Tools for Data Collection

Designing Tools for Programme Evaluation in Open and Distance Learning

Embed Size (px)

DESCRIPTION

Presentation at IGNOU workshop in 2007

Citation preview

Page 1: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

Evaluating a Programme in Distance EducationPreparing Tools for Data Collection

Page 2: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

Session Objectives

After this session, you are expected to be able to:• Identify appropriate tools for data collection; • List the characteristics of an appropriate tool;• Prepare tools for data collection to evaluate a programme

in Open and Distance Learning.

Page 3: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

What is Programme Evaluation?

• A systematic gathering, analysis and reporting of data about a programme to assist decision-making.

• A process of measuring the worth of a programme.

Page 4: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

Mesurement in Programme Evaluation

• What?

• How?

Learning, Impact of the programme, cost-effectiveness, opinions, etc.

Require some tools; quantification of data.

Page 5: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

Range of Data Collection Tools

Machine Recording of data (video, computer)

Standardized Tests

Questionnaires

Checklists

Interview Schedules

Direct Observation Sheets

Page 6: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

Characteristics of a Tool

• Validity:

• Reliability:

Does the tool measure what it is supposed to measure?

How consistent the tool is in measuring the phenomenon?

Page 7: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

Types of Questions in a Tool

• Behaviour/ Action-based:

• Attitude/ Opinion-based:

What the student does? Ownership of electronic gadgets, attendance in counselling.

• Classification-based:

What the student thinks? Writing of assignments, reading of SLM

What the student is? Age, gender, and othe demographic variables

Page 8: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

How Questions might be responded

• Open-ended/ Close-ended Questions• Types of Data and precision sought

Nominal/ Categorical, eg. RC/SC Ordinal, eg. Rank order of choice of

technology Interval, eg. Likert scale, assume equal

difference between two points in a scale Ratio, eg. Distance of SC from home, has an

absolute zero

Page 9: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

Steps in Preparation of Tools

1.Objectives of data collection2.Collection of an item pool3.Validation of the items4.Pre-test for reliability5.Modification and finalization

Page 10: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

Issues in Tool Preparation

• Consider the language and comprehension level of your respondents• Be brief and focused• Think about the flow of the questions (logical)• Avoid ambiguous words• Avoid questions with a negative in them

• Avoid leading questions, eg. Do you agree that assignments are necessary in Distance Education? Yes No

• Avoid loaded questions, eg. Do you prefer conventional education, because there is widespread corruption in distance education? Yes No

Page 11: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

Issues in Tool Preparation

• Do not suggest the range of acceptable response, eg. How many hours have you spent in reading this Block?(a) < 2 hours (b) 2-5 hours (c) 6-10 hours (d) 11-15 hours

• Avoid questions that are irrelevant, eg. At what time of the day you prefer to attend counselling?(a) Morning (b) Afternoon (c) Evening (d) Night

• Do not mix more than one question, eg. How do you find the counselling sessions helpful in understanding Course 01/02/04/05?(a) Excellent (b) Good (c) Satisfactory (d) Unsatisfactory

Page 12: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

What Tools to Use?

• A Programme Coordinator wants to gather feedback every year.

• A Course-writer/ Coordinator wants to check the instructional design quality of SLM

• An Assistant Regional Director wants to observe a counselling session/ practical for monitoring.

• An evaluator wants feedback on counselling sessions from counsellors.

• A Course Coordinator wants Expert Opinion on the quality of a course.

• An evaluator wants feedback on the quality of assignment evaluation and two-way communication.

Page 13: Designing Tools for Programme Evaluation in Open and Distance Learning

© Sanjaya Mishra, 2007

CreditsThanks to the Participants of this Workshop and earlier workshop on Programme Evaluation

Thanks to the University for giving such opportunities

Thanks to the Workshop Coordinator, Prof. Basanti Pradhan

A STRIDE-IGNOU PRESENTATION

THANK YOU