Integrating Quantitative and Qualitative Methods in

Evaluation

Brad Cousins, Ph.D.University of Ottawa

CaDEA Workshop Series, Session 4, Yaoundé, October 2010

Quantitative vs. Qualitative Methods

Quantitative QualitativeTypical goals Impact eval. Rich description

Ideal design RCT Ethnographic

Approach Pre-ordinate Emergent

Criteria Objectivity Depth of analysis

Reporting Impersonal Narrative

Strengths Replication / tests Grounded knowledge

Limitations Meaning, flexibility Replicability; labour intensive

Method Mixing Choices

Parallel: quantitative and qualitative methods are used simultaneouslySequential: one follows the other1. Qualitative followed by quantitative:

• Little known about domain of interest. • Helps to focus investigation, e.g., clarification of indicators

and measures• Test for generalizability of findings

2. Quantitative followed by qualitative• Clarification of findings through further in-depth inquiry

3

Justifications

Triangulation:– Corroboration of findingsComplementarily

– Added valueInitiation

– Discovering new constructsPractice observation:

– Evaluations fail to really capitalize on the power of multiple lines of evidence

4

A view of mixed methods inquiry (Greene, 2010)

The intentional, and connected or linked, use of more than one social science tradition, methodology, and/or method in service of better understanding– Tradition = philosophical paradigms and

assumptions, logics of justification, privileged questions, ways of knowing

Examples: postpositivism, interpretivism, constructivism, feminisms, critical social science

A view of MM, continued– Methodology = inquiry logic, including

questions, design, sampling, method choice, analysis, quality criteria, and defensible forms of writing

Examples: experimentation, survey research, ethnography, case study, narrative inquiry

– Method = a technique or tool for data gathering

Examples: Ask ~ questionnaire, interview, assessmentWatch ~ observationFind traces ~ unobtrusive measures

A view of MM, continued

A study is an MM study when there is some connection or linkage among the various methods and data sets at one or more stages of inquiry. Connection in MM can be at any stage in the inquiry, but is generally most common in the interpretation stage

Mixing methods at data analysis stage

Data cleaningData reduction and descriptionData transformationData correlation and comparisonAnalyses for inquiry conclusions and inferences

Integrated analyses – a few principles

Connected to prior methodological decisions.Highly iterative, spirit of adventure and discoveryNot every creative idea for interactive analyses will generate sensible or meaningful results. Cognitive activityPlanned stopping point or decision pointConvergence, consistency, and corroboration are overrated: dissonance as stimulus for inquiryChallenges to data quality and integrity can arise as a result of data transformation and framing

Integrated analysis, conceptual framework

Data transformation, enabling joint analyses

Data comparison and correlation, looking for patterns

Major analyses, leading to inferences and conclusions

Data transformation, one form to another (conversion)

Data consolidation or merging, multiple data sets into one

Data importationExtreme case analysisIntegrated data display

Warranted assertion analysisPattern matchingIntegrated data display

Selected readings

Greene, J. (2010, September). Snapshots of integrated analyses in mixed methods analyses. Paper presented at the Canadian Evaluation Society, National Capital Chapter, Ottawa.

Greene, J. (2009). Meaningfully Engaging with Difference through Mixed Methods Educational Evaluation. In K. E. Ryan & J. B. Cousins (Eds.), Sage International Handbook of Educational Evaluation. Thousand Oaks: CA: Sage.

Greene, J. C., Benjamin, L., & Goodyear, L. (2001). The merits of mixing methods in evaluation. Evaluation, 7, 25-44.