QUALITATIVEQUALITATIVEMETHODS METHODS

IN IN PROGRAMPROGRAM

EVALUATIONEVALUATIONBrian J. English Ph.D.Brian J. English Ph.D.

International Public Policy and Management ProgramInternational Public Policy and Management ProgramUniversity of Southern CaliforniaUniversity of Southern California

““Discovery consists in Discovery consists in seeing what everyone else seeing what everyone else

has seen and thinking what has seen and thinking what no one else has thought.”no one else has thought.”

Albert Szent-Gyorgi,Albert Szent-Gyorgi,1937 Nobel Prize1937 Nobel Prize

Physiology and Medicine Physiology and Medicine

Module ObjectivesModule Objectives What is qualitative research?What is qualitative research?

How do quantitative research and How do quantitative research and qualitative research differ?qualitative research differ?

What qualitative methods are What qualitative methods are appropriate for program evaluation?appropriate for program evaluation?

Why are qualitative methods Why are qualitative methods appropriate for gathering data to appropriate for gathering data to evaluate public policy and evaluate public policy and intervention programs?intervention programs?

Module ObjectivesModule Objectives

How can qualitative and quantitative How can qualitative and quantitative methods complement each other?methods complement each other?

How can qualitative data aid in the How can qualitative data aid in the planning and implementation of planning and implementation of intervention programs?intervention programs?

NOTE: This module will not cover data NOTE: This module will not cover data coding or the analysis of qualitative coding or the analysis of qualitative data. data.

Defining Qualitative ResearchDefining Qualitative Research

““A particular tradition in social science A particular tradition in social science thatthat

fundamentally depends on watching fundamentally depends on watching peoplepeople

in their own territory and interacting within their own territory and interacting withthem in their own language, on their ownthem in their own language, on their ownterms.”terms.”

(Kirk and Miller, 1986, p.9)(Kirk and Miller, 1986, p.9)

Defining Qualitative ResearchDefining Qualitative Research"Qualitative research is a loosely "Qualitative research is a loosely

defined category of research designs defined category of research designs or models, all of which elicit verbal, or models, all of which elicit verbal, visual, tactile, olfactory, and gustatory visual, tactile, olfactory, and gustatory data in the form of descriptive data in the form of descriptive narratives like field notes, recordings, narratives like field notes, recordings, or other transcriptions from audio- and or other transcriptions from audio- and videotapes and other written records videotapes and other written records and pictures or films.“and pictures or films.“

Judith Preissle Judith Preissle

Brian’s DefinitionsBrian’s Definitions

InterpretiveInterpretive

NaturalisticNaturalistic

Can be Ethnographic (a specific Can be Ethnographic (a specific

type)type)

DescriptiveDescriptive

Uses all 5 (or 6) of the sensesUses all 5 (or 6) of the senses

Writers’ research Writers’ research

How do Quantitative and How do Quantitative and Qualitative Research differ?Qualitative Research differ?

Qualitative researchers try to Qualitative researchers try to understand reality as it is construed understand reality as it is construed by the persons being studied.by the persons being studied.

Qualitative researchers try to Qualitative researchers try to understand the understand the meaning meaning of particular of particular activities or beliefs in the context of activities or beliefs in the context of the culture being consideredthe culture being considered

(Valadez and Bamberger, 1994)(Valadez and Bamberger, 1994)

QualitativeQualitative• InductiveInductive

• Subjective/Subjective/insider centeredinsider centered

• Process orientedProcess oriented

• Lack of control Lack of control of variablesof variables

QuantitativeQuantitative• DeductiveDeductive

• Objective/outsider Objective/outsider centeredcentered

• Outcome orientedOutcome oriented

• Attempt to control Attempt to control variablesvariables

QualitativeQualitative• Goal: Goal:

Understand Understand actors view actors view

• Discovery Discovery OrientedOriented

• ExplanatoryExplanatory

QuantitativeQuantitative• Goal: Find facts Goal: Find facts

and causesand causes

• Verification Verification OrientedOriented

• ConfirmatoryConfirmatory

Differences in DataFigure 1.1: Factors Influencing Pro-Active Environmental Behavior

KNOWLEDGE

ATTITUDINAL CHANGE

Religious Beliefs

Social Norms

Behavioral Change

Improved Environmental Conditions

$$$ Incentives

Peer Groups Level of ED

Legislation and Law Enforcement

Coastal Resource

Management Program

What Qualitative Research Methods What Qualitative Research Methods are appropriate for gathering data for are appropriate for gathering data for

Public Policy Program Evaluation?Public Policy Program Evaluation?

Document analysis of archival Document analysis of archival

recordsrecords

Observation & Participant Observation & Participant

ObservationObservation

Interviews/Focus groups Interviews/Focus groups

Chismoso & LaquacheroChismoso & Laquachero

TriangulationTriangulation

Document AnalysisDocument Analysis

NewspapersNewspapers Birth/death recordsBirth/death records Environmental Impact ReportsEnvironmental Impact Reports Financial ReportsFinancial Reports Previous EvaluationsPrevious Evaluations JournalsJournals Film & Photos (Grand Canyon Film & Photos (Grand Canyon

example)example)

““You can You can observe a lot observe a lot

just by just by watching."watching."

--Yogi--Yogi BerraBerra

ObservationObservation

Gaining AccessGaining Access

Participant ObservationParticipant Observation

Etic vs. EmicEtic vs. Emic

ShadowingShadowing

Interviews/Focus GroupsInterviews/Focus Groups

Key InformantsKey Informants

Confidential InformantsConfidential Informants

Informal InterviewsInformal Interviews

Structured InterviewsStructured Interviews

Chismoso & LaquacheroChismoso & Laquachero Talking and WalkingTalking and Walking

Need an Outgoing, Communicative Need an Outgoing, Communicative ApproachApproach

Gaining trustGaining trust

Note-taking & recordings can frighten Note-taking & recordings can frighten or turn-off potential informantsor turn-off potential informants

TriangulationTriangulation

Document AnalysisDocument Analysis

Interviews Interviews ObservationObservation

Why are qualitative methods Why are qualitative methods appropriate for gathering data to appropriate for gathering data to

evaluate public policy and evaluate public policy and intervention programs?intervention programs?

&&

How can qualitative and How can qualitative and quantitative methods quantitative methods

complement each other?complement each other?

The intricate relationship The intricate relationship between human behavior and between human behavior and public policies can restrict or public policies can restrict or promote specific behaviors.promote specific behaviors.

Examples:Examples: Restrictions on fishingRestrictions on fishing Prescription Drug policiesPrescription Drug policies Zoning LegislationZoning Legislation

The connections between public The connections between public policy, infrastructure planning, policy, infrastructure planning,

environmental degradation, environmental degradation, health, education and povertyhealth, education and poverty

East Coast Land DevelopersEast Coast Land Developers Waste Management and Dengue Waste Management and Dengue

FeverFever Coastal Resource ManagementCoastal Resource Management

Numbers do not always Numbers do not always reflect realityreflect reality

Ivy tower syndromeIvy tower syndrome

Glass ceiling syndromeGlass ceiling syndrome

US prisonsUS prisons

Money drives policy; Money drives Money drives policy; Money drives researchresearch

The data collected in qualitative The data collected in qualitative studies can be useful in studies can be useful in

developing quantitative studies.developing quantitative studies.

Determining IndicatorsDetermining Indicators

Identifying factors/variablesIdentifying factors/variables

Adding PerspectiveAdding Perspective

The learning process and The learning process and behavioral changes that take behavioral changes that take

place during a program place during a program intervention may best be intervention may best be

explored in a natural setting over explored in a natural setting over a long period of time.a long period of time.

Crosscheck information gathered in Crosscheck information gathered in quantitative surveysquantitative surveys

Aid in planning educational activitiesAid in planning educational activities Aid in Program revisionAid in Program revision

How can data aid in the How can data aid in the planning and implementation planning and implementation

of intervention programs?of intervention programs? Qualitative methods can reveal how Qualitative methods can reveal how

the local population feels about a the local population feels about a new project and how that project new project and how that project may affect their lives.may affect their lives.

Program designers have a moral Program designers have a moral

obligation to try to understand how obligation to try to understand how locals feel about interventions. locals feel about interventions.

StrengthsStrengths

• Depth & DetailDepth & Detail

• OpennessOpenness

• Insider Insider

PerspectivePerspective

• Tries to avoid Tries to avoid

pre-judgingpre-judging

• Can be less Can be less

expensiveexpensive

WeaknessesWeaknesses

• Smaller SampleSmaller Sample• Less easily Less easily

generalizedgeneralized• Difficult to make Difficult to make

systematic systematic comparisonscomparisons

• Dependant on Dependant on researchers skillsresearchers skills

• The effect of the The effect of the researcher’s researcher’s participationparticipation

Considerations for Further ThoughtConsiderations for Further Thought

UnobtrusivenessUnobtrusiveness Fitting inFitting in Ethical IssuesEthical Issues Legal IssuesLegal Issues CodingCoding Analysis Analysis Drawing InferencesDrawing Inferences ValidityValidity

Comments Comments &&

QuestionsQuestions