Semester 2: Lecture 9

Analyzing Qualitative Data: Evaluation Research

Prepared by: Dr. Lloyd Waller ©

Evaluation Research

Interventions are introduced to institutions, groups, organizations and communities usually with the goal of achieving certain goals and objectives or at least with the expectations for certain outcomes.

Interventions often need evaluation – mid-term evaluation, or Quarterly evaluation or end of programme evaluations - to determine whether or not it has achieved its goals.

Evaluations are normally conducted by doing Evaluation Research

Evaluation Research is a type of Research Methodology where research is undertaken or the purpose of determining the impact of intervention.

Evaluation Research

Political Science Research and Evaluation Research

• closely related to, but distinguishable from more traditional social research.

• utilizes many of the same methodologies used in traditional social research

• Requires special skills not usually taught in political science research:– group skills– management– political dexterity– sensitivity to multiple stakeholders

Evaluation Research

The Goals of Evaluation Research

• to provide "useful feedback“– sponsors– Donors– client-groups– Administrators– staff, and – other relevant constituencies

• to aid in decision-making

• to aid in policy-making

• to improve an intervention

• to promote change

Evaluation ResearchEvaluation Strategies

• Scientific-Experimental Models– Influenced by the social sciences - they prioritize on the desirability of impartiality,

accuracy, objectivity and the validity of the information generated

– Follows guidelines which are presumed by the research community

– Includes techniques such as:

• experimental research designs• quasi-experimental research designs• cost-effectiveness • cost-benefit analysis;

– Emphasis is on causal connections, emphasizing measurement as an integral part of evaluation.

– Quantifiable measures are critical to this type of evaluation, such as:• Psychometrics• Econometrics

Evaluation Research

Evaluation Strategies

• Management-Oriented Systems Models

– Primary emphasis is on comprehensive process management, with ongoing evaluation by management a key element

– Strategies include:• MBO (Management by Objectives)• PERT (Program Evaluation and Review)• CPM (Critical Path Method)• CIPP (Context, Input, Process, Product)• Information Auditing• Information Mapping• Logical Framework

Evaluation Research

Evaluation Strategies

• Participant-oriented – Emphasize the importance of the participants in the

process, especially the clients and users of the program or technology

– Focus on the claims, concerns, and issues of a variety of potential stakeholders:

• agents (implementers, funders) • beneficiaries (included target groups) • victims (excluded target groups)

Evaluation Research

Evaluation Strategies

• Qualitative/anthropological Models– These include the various naturalistic forms of

evaluation, emphasizing the value of subjective human interpretation of the observations made.

– Evaluator's role is to provide expert judgments on the outcomes of interest.

– Strategies include:• critical theory • grounded theory'

Evaluation Research

Types of Evaluation

• There are many different types of evaluations depending on the object being evaluated and the purpose of the evaluation– Formative evaluation – Summative evaluation.

Evaluation Research

Types of Evaluation

Formative evaluation – needs assessment determines who needs the program, how

great the need is, and what might work to meet the need – evaluability assessment determines whether an evaluation is

feasible and how stakeholders can help shape its usefulness – structured conceptualization helps stakeholders define the

program or technology, the target population, and the possible outcomes

– implementation evaluation monitors the fidelity of the program or technology delivery

– process evaluation investigates the process of delivering the program or technology, including alternative delivery procedures

• .

Evaluation Research

Types of Evaluation

Summative evaluation.– outcome evaluations investigate whether the program or technology

caused demonstrable effects on specifically defined target outcomes – impact evaluation is broader and assesses the overall or net effects --

intended or unintended -- of the program or technology as a whole – cost-effectiveness and cost-benefit analysis address questions of

efficiency by standardizing outcomes in terms of their dollar costs and values

– secondary analysis reexamines existing data to address new questions or use methods not previously employed

– meta-analysis integrates the outcome estimates from multiple studies to arrive at an overall or summary judgement on an evaluation question

Evaluation Research

Types of Evaluation

• Evaluation Questions and Methods– Formative research - major questions and methodologies

What is the definition and scope of the problem or issue, or what's the question?

• Formulating and conceptualizing methods might be used including brainstorming, focus groups, nominal group techniques, Delphi methods, brainwriting, stakeholder analysis, synectics, lateral thinking, input-output analysis, and concept mapping.

Where is the problem and how big or serious is it? • The most common method used here is "needs assessment"

which can include: analysis of existing data sources, and the use of sample surveys, interviews of constituent populations, qualitative research, expert testimony, and focus groups.

Evaluation Research

Types of Evaluation

• Evaluation Questions and Methods– Formative research - major questions and methodologies

How should the program or technology be delivered to address the problem?

• Some of the methods already listed apply here, as do detailing methodologies like simulation techniques, or multivariate methods like multiattribute utility theory or exploratory causal modeling; decision-making methods; and project planning and implementation methods like flow charting, PERT/CPM, and project scheduling.

How well is the program or technology delivered? • Qualitative and quantitative monitoring techniques, the use of

management information systems, and implementation assessment would be appropriate methodologies here.

Evaluation Research

Types of Evaluation

• Evaluation Questions and Methods– Summative Evaluation - major questions and methodologies

What type of evaluation is feasible? • Evaluability assessment can be used here, as well as standard approaches for

selecting an appropriate evaluation design.

What was the effectiveness of the program or technology? • One would choose from observational and correlational methods for

demonstrating whether desired effects occurred, and quasi-experimental and experimental designs for determining whether observed effects can reasonably be attributed to the intervention and not to other sources.

What is the net impact of the program? • Econometric methods for assessing cost effectiveness and cost/benefits would

apply here, along with qualitative methods that enable us to summarize the full range of intended and unintended impacts.

Evaluation Research

Types of Evaluation

• Evaluation Questions and Methods– Summative Evaluation - major questions and methodologies

What type of evaluation is feasible? • Evaluability assessment can be used here, as well as standard approaches for

selecting an appropriate evaluation design.

What was the effectiveness of the program or technology? • One would choose from observational and correlational methods for

demonstrating whether desired effects occurred, and quasi-experimental and experimental designs for determining whether observed effects can reasonably be attributed to the intervention and not to other sources.

What is the net impact of the program? • Econometric methods for assessing cost effectiveness and cost/benefits would

apply here, along with qualitative methods that enable us to summarize the full range of intended and unintended impacts.

• Input Measurement

• Output/Performance Measurement

• Impact/Outcomes Assessment

Evaluation Research

Evaluation Research

Steps in Evaluation Research• WHAT ARE THE PURPOSE AND GOALS OF THE PROGRAM?

• WHAT EVALUATION QUESTIONS DO I NEED TO ANSWER

• WHAT TYPE OF DESIGN WILL GIVE ME THE DATA I NEED (Experimental, Quasi-Experimental)

• WHAT TOOLS WILL GIVE ME THE DATA I NEED (Methods of Data Collection)

• HOW DO I COLLECT THE DATA (Procedure)

• HOW DO I ANALYZE THE DATA AND MAKE THE FINDINGS USEFUL