Justifying the adequacy of samples in qualitative

interview-based studies: Differences between and

within journals

Prof Julie Barnett a, Konstantina Vasileiou a, Dr Susan Thorpe b, Prof Terry Young c

a University of Bath, Department of Psychology

b Newcastle University, School of Psychology

c Brunel University London, College of Engineering, Design and Physical Sciences

Symposium: “Quality in qualitative research and enduring problematics”

Qualitative Methodology Forum – 27 January 2015

Faculty of Humanities and Social Sciences, University of Bath

Experiential triggers for this project (1)

“First of all, most of the

articles published in the

journal are not qualitative in

protocol. So we need to

enter this manuscript gently

as has been done with

some others”

Editor

“Third, I don’t have a good understanding of the

representativeness of the sample. Only 30% of

winning organizations were represented. Of them,

the key informants are not at all well-described.

How do we know that these 15 people best

represent their organizations? Who are they?

What are their characteristics. The sampling issue

(at both of two levels, organization selection and

individual selection within organizations) is a

critical issue”

“I thought for a topic like this you may need to increase

the number of participants to at least 25 (50%). This was

not a random selection. 15 is respectable. However,

can you prove they are representative of the 51

winners? Secondly what are these winners of? What

are the broader implications of using such subjects?

Who can they speak for?”

Experiential triggers for this project (2)

“While no decisions

regarding clinical practice

should ever be based on

such a small sample size,

in the end I believe it makes

a contribution solely in the

way we conceptualize

evidence. “

Reviewers

“Your sample size of 15 out of 51 organizations

is very small….However, the issue of sampling

error is just as pertinent to qualitative researchers

as it is to quantitative researchers. Three

statements are just not enough for me to believe

that the result will generalize to the larger

population. In other words, how do I know this is a

real signal rather than noise? The generalizability

of the results, perhaps, might be more believable if

a large proportion of the interviewees concurred

with the quoted statements.”

“There remains an issue of

how useful this information

would be. The poor

response rate is certainly

something that gives the

reader pause, and the

results of the work cannot

be seen as generalizable.”

“The generalizability of findings is what

makes science different from faith-based

study. If the purpose of qualitative

research is not to produce findings that

are capable of generalizing beyond the

data gathered, then why should a reader

care about the results? We only care

when we think the results may

generalize to our own studies,

theories, situations, etc.”

How many qualitative interviews are enough? Purposeful sampling → Informationally representative samples (vs statistically

representative)

Tension between achieving informational redundancy and be able to conduct in-depth,

‘case-oriented’ analysis (Sandelowski, 1995)

So how many interviews are enough?

Experts in the field tend to concur with the answer that “It depends…” (Baker & Edwards,

2012)

Research objective(s)

Epistemological and theoretical underpinnings

Type of analysis

Epistemic community (i.e. rules, norms of the scientific community one belongs to)

Practical considerations (e.g. hard to access participants; resources; researcher’s career

stage; institutional constraints)

The criteria of data/empirical saturation or theoretical saturation (i.e. when no new

insights about the phenomenon/theoretical category are found by adding new cases) remain

useful in determining qualitative sample sizes.

Our Research Questions

To what extent are arguments to justify the sample size

of qualitative interview-based research employed?

What are these arguments?

Do the presence and/or the nature of justifications differ

across journals from different disciplines?

How might the justifications relate to other

characteristics such as the type of analysis?

What justifications, other than sample size, are used to

defend the adequacy of the sample?

Do the various justifications change over time?

How do we ‘do’ science: Scrutinising

published research

A few examples…

How is theory used in qualitative research? (Bradbury-Jones, C.,

Taylor, J., Oliver Herber, O. (2014). How theory is used and articulated in qualitative research:

Development of a new typology. Social Science & Medicine,120, 135-141)

How are research questions constructed in social

scientific work? (Alvesson, M., & Sandberg, J. (2013). Constructing research

questions: doing interesting research. London: Sage)

Are participant recruitment and retention in RCTs

adequately reported? (Toerien et al. (2009). A review of reporting of participant

recruitment and retention in RCTs in six major journals. Trials, 10, 52.)

Our Methods

• Systematic review of qualitative interview-based

studies

• Published between Jan 2003 and Dec 2013 in high

quality healthcare-related journals representing different

disciplines

Journals:

British Medical Journal (BMJ) (Medical focus)

British Journal of Health Psychology (Psychology)

Sociology of Health & Illness (Sociology)

Journal of Healthcare Management (Management Sciences)

Social Science & Medicine (Interdisciplinary Social Sciences

journal)

Inclusion/exclusion criteria

Cross-sectional study design (i.e. longitudinal studies were

excluded)

Individual, qualitative interviews as method of data collection (i.e.

group interviews and structured interviews were thus excluded)

Data analysed qualitatively (i.e. studies that quantified their

qualitative data were excluded)

Mixed method studies were excluded (e.g. qualitative interviews

and structured questionnaires)

Papers reporting more than one qualitative methods of data

collection were excluded (e.g. individual interviews and focus

groups)

Data Extraction Form

Data analysis

We used both qualitative and quantitative analysis of data

Some preliminary results: British Medical

Journal (BMJ)

Search keywords: interview* AND qualitative

132 results were obtained from the search

All 132 results were screened as to

whether they met the criteria

22 eligible articles were included in the

review

Identification

Screening

Data were extracted from the 23 articles

that met the eligibility criteria Eligibility

Included One paper was excluded

due to longitudinal design

BMJ: Some basic stats

Countries of data

collection

Frequency

of papers

Percentage

%

UK 16 72

Netherlands 2 9

Canada 1 4

Serbia 1 4

Australia 1 4

South Africa &

Uganda

1 4

Total 22 100.0

N of Interviews N of participants

Minimum 19 19

Maximum 128 128

Median 31 30.5

Mean (SD) 44.14 (31) 44.73 (31.05)

Table 1

Countries of data collection

Table 2

N of interviews conducted and N of participants included

Study Populations N of papers involved

each study population Percentage (%) of the

total number (N =22) of

studies*

Patients 13 59 Doctors 5 22 Nurses 3 14 Relatives & Significant others 3 14 Senior Healthcare Managers 1 4

Healthcare Administrative

Staff 1 4

Children (but not patients) 1 4 Caregivers 1 4

Other (e.g. sex workers, ex-

offenders etc.) 4 18

Table 3

Study populations

* Some studies involved more than one study population (e.g. patients and doctors).

BMJ: Justification of sample size

The majority of papers (N = 12; 54.5%) did not justify their

sample size in any way

Types of Justification Frequency Percentage % over the total

number of justifications

provided

Data saturation 7 41

Theoretical Saturation 2 12

Previous literature 2 12

Sample pre-defined requirements

(e.g. maximum variation sampling)

2 12

Pragmatic reasons 2 12

Researchers’ experience 1 6

Nature of qualitative data 1 6

Total 17 100.00

For the papers that

DID justify their

sample size (N = 10;

45.5%)

Number of different

justifications

provided by

‘justifying’ papers (N

= 10)

BMJ: Qualitative analysis of sample adequacy

argumentation (1)

Diversity/Variation

The argument of diversity counterbalanced the lack of sample

representativeness

Study Populations

“We…included the views not

only of patients and GPs but

also of practice nurses and

receptionists, who have

generally been excluded from

previous studies” (BMJ02)

Participant Demographic characteristics

“One strength of our study was the diverse

range of respondents in terms of age,

socioeconomic group, location, and household

smoking profile.(BMJ10)

“Participants were purposely sampled to

represent a wide range in medical specialties,

age, and sex to reflect the possible diversity of

opinions.” (BMJ19)

Aspects of the phenomenon of

interest

“An additional strength is its focus on

reactions to intermediate results as

well as positive and negative

diagnostic results” (BMJ07)

“As with any qualitative study aiming for a maximum

variation sample, the findings are not intended to be

numerically representative— the sampling method is

intended to show the diversity in responses, including

those that are less usual. (BMJ17)

“These tables were, however, derived from a purposive

sample and should not be taken to represent the

population; rather, we aimed to capture the range and

diversity of experience, beliefs, and opinions instead of

providing a quantitative summary of findings.” (BMJ20)

BMJ: Qualitative analysis of sample adequacy

argumentation (2)

Sample particularity: Constructed negatively as it undermines the

potential of representativeness of the sample and thus limits the

generalizability of findings

“The generalisability of our findings is

limited because the sample was drawn

from a prison in southwest England that

predominantly holds white British

offenders with sentences of less than one

year; to what extent our findings might

relate to long term offenders, those from

black and ethnic minorities, or women is

therefore unclear.” (BMJ09)

But because of the particular

nature and characteristics

pertinent to older people and

patients with chronic pain the

results presented may not be

generalisable to other drugs

or different age groups.”

(BMJ11)

“One limitation of our study is that we looked only

at cases in which a request for euthanasia had

not been granted or granted but not performed

(about two thirds of all requests), and the

perspectives of patients and physicians with

regard to unbearable suffering might be different

in cases where euthanasia was performed—for

instance, showing more agreement between

patients and physicians.” (BMJ15)

“Finally, within the sub-Saharan African

population, the participants in this study

are a relatively unusual group in that all

were receiving palliative care. Most people

dying with advanced illness in South Africa

and Uganda, and indeed in the rest of sub-

Saharan Africa, may have even less

access to information and care than this

sample.” (BMJ16)

"Another limitation of our study is

that we focused on “difficult” and

“straightforward” cases rather than

on the type of cancer; our study

may therefore not be

representative for the whole

cancer population.” (BMJ19)

Some preliminary conclusions…

Just over half of the papers did not justify their sample size in any

way

Claim to data saturation was the most common justification for the

sufficiency of sample size

Stakeholder, demographic or phenomenal diversity/variation was

the strongest line of defence of sample adequacy

Sample particularity was viewed to be problematic as it undermined

the scope of sample and thus the generalizability of results

There was an absence of claims around ‘theoretical

generalisations’

Quasi-quantitative referential system of evaluation of

sample adequacy

Thank you for listening!

Any questions?

References

Alvesson, M., & Sandberg, J. (2013). Constructing research questions: doing interesting research.

London: Sage

Baker, S.E., & Edwards, R. (2012). How many qualitative interviews is enough? Expert voices and

early career reflections on sampling and cases in qualitative research. National Centre for

Research Methods Review Paper.

Bradbury-Jones, C., Taylor, J., Oliver Herber, O. (2014). How theory is used and articulated in

qualitative research: Development of a new typology. Social Science & Medicine,120, 135-141

Sandelowski, M. (1995). Sample size in qualitative research. Research in Nursing & Health, 18, 179-

183.

Toerien et al. (2009). A review of reporting of participant recruitment and retention in RCTs in six

major journals. Trials, 10, 52.