Publication bias in health service delivery research

15/04/2023

Yen-Fu Chen, Richard LilfordWarwick Centre for Applied Health Research & Delivery (W-CAHRD)

University of Warwick

CLAHRC West Midlands International Scientific Advisory Group

Publication and related bias

• Publication bias

The tendency on the parts of investigators, reviewers, and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings (Dickersin 1990)

• Dissemination bias

- Outcome reporting bias

- Citation bias

- Language bias

- Media attention bias• P-hacking

Publication bias in biomedical literature (and beyond)

• Well established in biomedical research e.g. Song et al. Health Technol Assess 2010;14(8)

• Led to the creation and enforcement of clinical trial registries

• Schmucker et al. PLoS ONE 2014;9(12): e114023

- REC: 46% (prediction interval 22% to 72%) published

- Trial registries: 54% (prediction interval 13% to 90%) published

• Increasing evidence in other scientific disciplines: social sciences, management, economics, ecology and evolution

Publication bias in health services & delivery research (HSDR)

• Scoping search identified very limited literature‘publication bias’ and key terms such as ‘health services research’ or ‘health services management’ or ‘service delivery’ or ‘quality improvement’ or ‘patient safety’

• Why documentation of empirical evidence is lacking?- Not an issue in HSDR- Lack of awareness- Choose to ignore- Difficult to locate- Difficult to study

Nature of HSDR

• Definition and boundary

- difficulties in searching the literature• Multiple variables and relationship between them• Diverse research methods

Publication and related bias in HSDR: study design

Experimental studies (e.g. RCTs)

Quasi-experimental studies (e.g. natural experiments, instrumental variable analyses, ITS, CBA)

Non-experimental (observational) studies(e.g. uncontrolled before-and-after study, analysis of routine data)

Quality improvement projects

ResearchPublication bias?

Scientific enquiry

Practical instrument

Methods for investigating publication bias

• Direct evidence

1. Following a cohort of studies over time

2. Survey of researchers• Indirect evidence – examining the literature

3. Assessing publication and related bias in systematic reviews

4. Methodological (case) studies to explore methods for detecting and/or mitigating publication and related bias

• Empirical evidence in HSDR & proposed research

• Time-sharing Experiments in the Social Sciences (TESS)• Peer-reviewed, competitive process; surveys conducted by the same

research firm

Empirical evidence 1:inception cohort of HSDR studies

Franco et al. Science 2014;345:1502-5

Null (n=48)

Mixed (n=82)

Strong (n=91)

Not written 65% 12% 4%

Written but not published 15% 39% 34%

Published (non-top-tier) 10% 38% 38%

Published (top-tier) 10% 11% 23%

2 (6) = 80.3, P < 0.0001

• Review of the effects of mass mailings on influenza vaccination uptake among Medicare beneficiaries

• Medicare Peer Review Organization Health Care Quality Improvement Project (HCQIP) database

• Six controlled trials identified• Only one study reporting modest but statistically significant results

was published (increase of 8.7% vs 6.5% vs 4.4%)• The remaining did not observe clinically meaningful improvement –

none published

Empirical evidence 1: inception cohort of HSDR studies

Maglione et al. Am J Prev Med 2002;23(1):43–46

Proposed research 1: Following an inception cohort of HSDR studies over time

• Identification of a suitable cohort

- Lack of comprehensive, accessible registries for HSDR

- Ethics committees

- Papers presented in conferences

- Manuscripts submitted to journals• Existing evidence suggests non-submission by

investigators is the main cause of non-publication

Empirical evidence 2: Interrogation of HSDR stakeholders

1. Planned or in preparation “May publish following validation”2. Not of interest for others (generalizability too limited)“Constellation of internal social factors, adoption factors, staff training/experience, etc. seemed too unique to make it general enough”3. No time for writing Funding ran out; or new projects started; “Too busy implementing CPOE to publish” 4. Limited scientific quality“The setup (e.g., amount of interviews) was not robust enough” 5. Political and legal reasons “Government was unwilling to publicly share negative content of initial responses”6. Only meant for internal use “The evaluation was only meant for the own organization; academic output is not necessary”

Ammenwerth & De Keizer, J Am Med Inform Assoc. 2007;14:368–371

Proposed research 2: Interrogation of HSDR stakeholders

• Gather perceptions and first-hand experience of relevant stakeholders

• Selection of study sample- Who (academics, policy makers? service managers?)

- What discipline to include

• Questionnaire or interview or focus group

Empirical evidence 3: Published survey of systematic reviews

9%

23%

68%

Assessment of likelihood of publication bias in Cochrane EPOC reviews (n=99)

Yes Unclear NoLi et al. Health Policy 2015; 119:503–510

Yes: explicitly assessed

Unclear: partial information, e.g. discussed in conclusion

No: not assessed for some reason or no information

Empirical evidence 3: Published survey of systematic reviews

• Overview of systematic reviews of interventions for improving care for patients with diabetes

• Identified 125 reviews, of which 50 higher quality reviews further assessed

• 22/50 (44%) assessed likelihood of publication bias

Worswick et al. Syst Rev 2013;2:26

Proposed research 3:Survey of published HSDR systematic reviews

• Sampling Cochrane EPOC and other published HSDR systematic reviews

• Reasons for not assessing publication bias- Too few studies- Narrative / qualitative synthesis only- Perceived heterogeneity- Something else?

• Where the likelihood of publication bias was assessed:- Methods used- Findings

Empirical evidence 4: Methodological case studies

Health care utilisation

Costa-Font et al. Health Policy 2013;109: 78– 87

“The winner’s curse”

• Identification and selection of cases- Experimental vs non-experimental

- Intervention effectiveness vs association

• Candidate topics- QI interventions for care of patients with diabetes

- Computerised decision support systems

- Audit & feedback

- Weekend admissions & mortality

- Volume & outcomes

- Staffing ratio & outcomes

• Methods to explore- Funnel plot

- Regression (precision, impact factor etc.)

- p-curve?

Proposed research 4:Case studies of applicability of common and emerging methods

Head et al. PLoS Biol 2015;13(3): e1002106

p-curve

Thank you & acknowledgement

Critique• Professor Russell Mannion• Professor John Øvretveit