Judging Qualitative Research

The Role of the Reader "There are no operationally defined truth tests to apply to

qualitative research" (Eisner, 1991, p. 53).

Researcher and readers "share a joint responsibility" for establishing the value of the qualitative research product (Glaser and Strauss, 1967, p. 232).

"Pragmatic validation [of qualitative research] means that the perspective presented is judged by its relevance to and use by those to whom it is presented: their perspective and actions joined to the [researcher’s] perspective and actions" (Patton, 1990, p. 485).

Validity of research corresponds to degree to which it is accepted as sound, legitimate and authoritative by people with an interest in research findings.

How do we judge which perspective to use to evaluate the validity of a qualitative study? E.g., grounded theory should theoretically sample a wide

range of people; discourse analysis can be in-depth analysis of a few excerpts

Process of agreeing criteria for judging qualitative research is useful because involves critically reflecting on essential ingredients & practices

But simply following guidelines does not guarantee good research – not simply a descriptive science but relies on capacity to evoke imaginative experience & reveal new meanings.

3 features Coherence (structural corroboration or

triangulation)

Consensus

Instrumental Utility "Guides call our attention to aspects of the situation

or place we might otherwise miss" (Eisner, 1991, p. 59

Trustworthiness"How can an inquirer persuade his or her audiences

that the research findings of an inquiry are worth paying attention to?" (Lincoln and Guba 1985, p. 290).

Lincoln and Guba (1985, p. 300) - an alternative set of criteria that correspond to those typically employed to judge quantitative work.

Comparison of criteria

Conventional terms Naturalistic terms

Internal validity Credibility

External validity Transferability

Reliability Dependability

Objectivity Confirmability

Critical of use of "comparable criteria” little different than the conventional criteria

assumes that what is known — existent or interpreted reality — stands independent of the inquirer and can be described without distortion by the inquirer

naturalistic research can offer only an interpretation of the interpretations of others

to assume an independent reality is unacceptable for many qualitative researchers

e.g., Smith & Heshusius 1986 there is no "out there" out there: the only reality is

a completely mind-dependent reality, which will vary from individual to individual;

therefore, not possible to choose a best interpretation from among the many available, because no technique or interpretation can be "epistemologically privileged" (p. 9).

This stance prohibits the possibility of reconciling alternative interpretations

Important to determine which criteria are consistent with the naturalistic paradigm, yet which allow for a declaration that good research has been carried out.

Select appropriate criteria for judging overall trustworthiness of a qualitative study

Internal Validity vs. CredibilityConventional inquiry Naturalistic inquiry= Extent to which findings accurately describe reality.

Must postulate relationships and then test them.

Postulate cannot be proved, only falsified.

= Assume the presence of multiple realities and try to represent these multiple realities adequately.

Isomorphism is in principle impossible: if we knew precise nature of reality, no need to test it.

Credibility becomes the test for this.

Credibility-A ‘toolbox’ of procedures for enhancing validityTriangulation: Enrich understanding of a phenomenon by

viewing from different perspectives. 4 types: 1) methods triangulation; 2) data triangulation; 3) triangulation through multiple analysts;4) theory triangulation.

gather data from different groups of people; gather data at different times from same people; different theories/methods ‘composite analysis’; triangulate perspectives of different researchers.

Comparing researchers’ coding Inter-rater comparison of subtle, complex coding

schemes; participant feedback/respondent validation

Making segments of the raw data available for others to analyze, and the use of "member checks," in which respondents are asked to corroborate findings.

Disconfirming case analysis Deviant or negative cases, systematically

searching for data that does not fit themes & patterns

Audit (paper) trail Provide evidence linking raw data to final report

Audit trail Consisting of 1) raw data; 2) analysis notes; 3) reconstruction and synthesis products; 4) process notes; 5) personal notes; and 6) preliminary developmental information

*Critical component = conceptual chain of logic - mimics replicability process in conventional research.

External Validity / Generalizability versus Transferability

Conventional: seek findings which are statistically generalisable

Naturalistic: aspire to theoretical, vertical or logical generalization of findings

= Ability to generalize findings across different settings.

Involves a trade-off between internal & external validity (can incl. only limited aspects of each local context).

= transferability of a working hypothesis to other situations depends on the degree of similarity between the original situation and the situation to which it is transferred.

Existence of local conditions makes it impossible to generalize.

Reliability versus DependabilityConventional Inquiry Naturalistic Inquiry

(Focus instead on achieving greater validity in work)

=

1) the degree to which a measurement, given repeatedly, remains the same;

2) the stability of a measurement over time;

3) the similarity of measurements within a given time period

= use an inquiry audit in which reviewers examine both the process and the product of the research for consistency.

"Since there can be no validity without reliability (& thus no credibility without dependability), a demonstration of the former is sufficient to establish the latter" (Lincoln & Guba 1985: 316).

Objectivity versus ConfirmabilityConventional Inquiry Naturalistic Inquiry

Relies on quantitative measures to define a situation is relatively value-free, & therefore objective.

Subjectivity leads to unreliable & invalid results.

Relies on interpretations and is admittedly value-bound, is therefore considered to be subjective.

Question the possibility of ever attaining pure objectivity.

Empathic Neutrality Patton (1990): the terms objectivity and subjectivity have

become "ideological ammunition in the paradigms debate." Use instead "empathic neutrality" (p. 55). Empathy "is a stance toward the people one encounters, while

neutrality is a stance toward the findings" (p. 58). A researcher who is neutral tries to be non-judgmental, and strives to report what is found in a balanced way.

Lincoln and Guba (1985): the "confirmability" of the research - the degree to which the researcher can demonstrate the neutrality of the research interpretations, through a "confirmability audit."

Demonstrating validity Sensitivity to context: allow patterns &

meanings NOT prespecified to emerge Relevant theoretical & empirical literature Socio-cultural setting Participants’ perspectives Ethical issues Empirical data

Sensitivity to the context of existing theory & research in the development of a research topic

Study lacks features of validity

Study sets out to simply ‘explore women’s experiences of postnatal depression’, ignoring relevant theory & previous qualitative studies of experiences of maternity & depression in women

Study demonstrates features of validity

Study clarifies what is already known form theory & research, formulates a specific research question that has not been addressed: “How does the family context influence women’s experiences following childbirth?”

Sensitivity to how the perspective & position of participants may influence whether they feel able to take part & express themselves freely

Senior male professional carries out interviews with women with postnatal depression in a clinic setting, ignoring possibility they may feel less able to express feelings to a man, my be intimidated by clinical environment.

Participants give choice of taking part in focus groups (allowing solidarity with pother women in similar experiences) or interviews in own home (privacy, security, accessibility) carried out by women of own age.

Commitment & rigour through data collection Depth/breadth of analysis methodological competence/skill In-depth engagement with topic

Commitment & rigour in recruitment of participants who will represent an adequate range of views relevant to research topic

Sample comprises 12 self-selected volunteers. Most are well-educated, affluent, married.

12 women are purposively sampled to include married, co-habiting & single participants from affluent and socio-economically disadvantaged backgrounds.

Coherence & transparency clarity & power of argument fit between theory and method transparent methods & data presentation Reflexivity

Transparency in the analysis of data

Little description is provided of how themes were identified and no checks on their consistency reported

A detailed description is provided of how data were initially coded and how codes were modified through comparison of all instances and discussions between the researchers.

Coherence between the qualitative design & the analysis and presentation of data

Based on a frequency count of the occurrence of codes, the researcher highlights the finding that 2/3 of the single women (n=3) but only 1/3 of married women complain of lack of social support. Strong quantitative statements of this kind are inappropriate given small sample size and reliability of codes is unknown.

Based on a qualitative comparisons of all instances of the codes, the researchers note that there was a tendency for single women to report a lack of social support. However disconfirming instances are discussed as revealing examples of why married & cohabiting women may feel unsupported and how single mothers may be supported by others.

Impact and Importance practical/applied theoretical socio-cultural

Impact of the research The researcher simply notes that the findings are compatible with existing models and research (e.g., showing that single mothers feel they have less social support).

The researchers explain how different family structures and relationships may exert positive or negative influences on the experience of maternity, suggesting questions for further research and ways of identifying and supporting women at risk of depression.

Not always possible to meet some criteria, sometimes must prioritize some kinds of validity over others

A set of principles to refer to when making decisions about how to carry out & to justify your research.

Reading

Willig Chapter 9.

Barbour, R.S. (2001) Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? British Medical Journal, 322:1115-1117.

Cho, J. & Trent, A. (2006) Validity in qualitative research revisited. Qualitative Research, 6, 319-340.

Harré, R. (2004) Staking our claim for qualitative psychology as science. Qualitative Research in Psychology, 1, 3-14.

Henwood, K. (2004) Reinventing validity. In Todd, Z., Nerlich, B., McKeown, S. & Clarke, D.D. (Eds.). Mixing methods in Psychology: The Integration of Qualitative and Quantitative Methods in Theory and Practice. Psychology Press. Chapter 3.

Parker, I. (2005) Qualitative Psychology: Introducing Radical Research. OUP. Chapter 10.