Systematic Review Module 8: Assessing Applicability C. Michael White, PharmD, FCP, FCCP Professor and Director University of Connecticut/Hartford Hospital

Systematic Review Module 8: Systematic Review Module 8: Assessing ApplicabilityAssessing Applicability

C. Michael White, PharmD, FCP, FCCPC. Michael White, PharmD, FCP, FCCP

Professor and DirectorProfessor and DirectorUniversity of Connecticut/Hartford HospitalUniversity of Connecticut/Hartford Hospital

Evidence-based Practice CenterEvidence-based Practice Center

Speaker has no actual or potential conflicts of interest in relation to this activity

Learning ObjectivesLearning Objectives

The successful learner will be able to:– Describe applicability and substantiate its

importance

– Delineate a systematic approach to assessing applicability Based on PICOTS domains

– Apply a standard approach to discerning whether a study is evaluating efficacy or effectiveness

2

Applicability of StudiesApplicability of Studies

Prepare topic:

· Refine key questions

· Develop analytic frameworks

Search for and select

studies:

· Identify eligibility criteria

· Search for relevant studies

· Select evidence for inclusion

Abstract data:

· Extract evidence from studies

· Construct evidence tables

Analyze and synthesize data:

· Assess quality of studies

· Assess applicability of studies

· Apply qualitative methods

· Apply quantitative methods (meta-analyses)

· Rate the strength of a body of evidence

Present findings

3

Defining ApplicabilityDefining Applicability

Applicability definition– “Inferences about the extent to which a causal

relationship holds over variations in persons, settings, treatments, and outcomes”

Applicable study results likely reflect expected outcomes in the real world

Others terms used synonymously with applicability include external validity, generalizability, and relevance

Shadish W, Cook. T Experimental and quasi-experimental design for generalized casual inference. Boston: Houghton Mifflin; 2002.

~Shadish and Cook, 2002

4

Framing Applicability IssuesFraming Applicability Issues

Frame issues of applicability with reference to specific clinical or policy questions the review is intended to inform

Applicability needs to be considered at the outset– When scope of review is determined

– When key questions are identified

Atkins D. Assessing applicability. Methods guide.

5

Applicability ResourcesApplicability Resources

Clinical experts and stakeholders can provide general information important in framing applicability issues – What the population of interest looks like

Mostly female, mostly elderly, mostly ethnic

– What types of care or procedures are routine or represent standard of care

– Are certain subpopulations characteristically different from others Biologically, clinically


6

Other Applicability Other Applicability ResourcesResources

Registry or epidemiological information, practice guidelines, consensus papers, book chapters, and general reviews can provide useful applicability information– Applicability issues do not have to be

reviewed for each study

– Used to place the available literature in context Should be a factor in rating the strength of

evidence


.

7

General Considerations in General Considerations in Judging ApplicabilityJudging Applicability

Applicability judgments should be based on stepwise considerations of a number of specific issues– However, applicability is a general rather

than absolute construct No validated formulaic criteria

Atkins D. Assessing Applicability. Methods Guide.

8

General Considerations in General Considerations in Judging ApplicabilityJudging Applicability

Stepwise approach to applicability:– Consider applicability based on nature of

interventions and outcomes

– Identify a few factors that are most relevant to applicability

– Summarize findings in a consistent way using PICOTS framework

– Summarize reasoning behind judgments made about applicability to other populations or interventions


9

Population and ApplicabilityPopulation and Applicability

Data to Abstract Conditions That Limit Applicability

Eligibility criteria, proportion of screened individuals enrolled

Narrow eligibility criteria, high exclusion rate

Demographics (range and mean): age, gender, race, ethnicity

Differences between study population and patients in community

Severity or stage of illness (referral or primary care population)

Narrow or unrepresentative severity or stage of illness

Run-in period: attrition rate before randomization and reasons (nonadherence, side effects, no response)

Run-in periods with high exclusion rates

Event rates in treatment and control groups

Events rates markedly different than in community

Prevalence of disease (for diagnostic studies)

Disease prevalence in study population different than in community

Atkins D. Assessing Applicability. Methods Guide. Gartlehner G. J Clin Epidemiol 2006;59:1040-8.

10

Population and Applicability: Population and Applicability: ExamplesExamples

In the FIT trial, only 4,000 of 54,000 women screened were enrolled. Women were younger, healthier, and more adherent than typical osteoporosis patients.

Trial of etanercept for juvenile diabetes excluded patients with side effects during an active run-in period. Trial found low incidence of adverse events.

Clinical trials used to inform Medicare decisions enrolled patients who were younger (60 vs. 75 years) and more often male (75 vs. 42%) than Medicare patients with cardiovascular disease.


11

Intervention and ApplicabilityIntervention and Applicability


Medication dose, schedule, duration

Regimen not reflective of current practice

Intensity of behavioral interventions Intensity of intervention not feasible for routine use

Adherence interventions Monitoring practices or visit frequency not used in practice

Version of rapidly changing technology

Versions not in common use

Cointerventions Cointerventions that likely modify effectiveness of therapy

Training/skill level of intervention team (surgery/diagnostics)

Level of training not widely available


12

Intervention and Intervention and Applicability: ExamplesApplicability: Examples

Studies of behavioral modification to promote healthy diet employ larger number and longer duration of visits than those available to most community patients.

Antiretroviral trials’ use of pill counts does not always translate into effectiveness in real-world practice.

Combining iron and zinc attenuates the ability of iron to raise hemoglobin levels.

Trials of carotid endarterectomy selected surgeons with extensive experience and low complication rates were not representative of average vascular surgeons.


13

Comparator, Outcomes, and Comparator, Outcomes, and ApplicabilityApplicability

Data to AbstractConditions That Limit Applicability

ComparatorMedication dose, schedule, duration (if applicable)

Regimen not reflective of current practice

Comparator chosen versus others available (if applicable)

Use of substandard alternative therapy

OutcomesOutcomes (benefits AND HARMS) and how they were defined

Surrogate endpoints, improper definitions for outcomes, composite endpoints


14

Comparator and Comparator and Applicability: ExamplesApplicability: Examples

Fixed-dose study compared high dose duloxetine (80 to 120 mg) to low dose paroxetine (20 mg)

Many trials evaluating magnesium in acute myocardial infarction conducted before thrombolytics, antiplatelets, beta-blockers, and primary percutaneous coronary intervention (PCI) were used

Only 1 of 23 trials comparing bypass surgery to PCI used drug-eluting stents


15

Outcomes and Applicability: Outcomes and Applicability: ExamplesExamples

Trials of biologics for rheumatoid arthritis use radiographic progression rather than symptom evaluations

Trials comparing cyclooxygenase-2 inhibitors and nonsteroidal antiinflammatory drugs use endoscopy-evaluated ulceration rather than symptomatic ulcers


16

Timing, Setting, and Timing, Setting, and ApplicabilityApplicability


TimingTiming of follow-up/outcome measures

Followup too short to detect important benefits or harms, measuring effects at inappropriate times

SettingGeographic setting Settings where standards of care differ

markedly from setting of interest

Clinical setting Specialty population or level of care that differs from community


17

Timing and Applicability: Timing and Applicability: ExamplesExamples

Alzheimer’s disease trials evaluate surrogate end points (cognitive function scales) at 6 months, which may not reflect long-term outcomes (institutionalization rates)

Trials evaluate the QTc interval-prolonging effects of drugs using single dose, end-of-dosing interval evaluations rather than evaluations at maximum blood concentrations


18

Setting and Applicability: Setting and Applicability: ExamplesExamples

Studies evaluating benefits of breast self-exams conducted in Shanghai and St. Petersburg, countries that do not employ routine mammography screening as in US– Would self-exam be as effective if routine

mammogram picks up cancer at earlier stages?

Studies of open surgical abdominal aortic aneurysm repair found inverse relationship between hospital volume and short-term mortality


19

Efficacy or EffectivenessEfficacy or Effectiveness

Seven criteria used– 5 of 7 indicative of effectiveness trial

1. Enrolled primary care population

2. Less stringent eligibility criteria

3. Assessment of health-related outcomes

4. Long study duration, clinically relevant treatment modalities

5. Assessment of adverse events

6. Adequate sample size to assess minimally important difference for a patient perspective

7. Intention to treat analysis

Gartlehner G. Int J Tech Assessment Health Care 2009;25:323-30. Gartlehner G. J Clin Epidemiol 2006;59:1040-8.

20

Assessment of Effectiveness Assessment of Effectiveness Decision ToolDecision Tool

EPC directors reviewed 26 trials – 20 were judged subjectively as

effectiveness trials, 6 as efficacy Scale not used

Using the scale, 17 of 20 met five criteria and only 1 of 6 efficacy trials did


21

Guidance for Assessing Guidance for Assessing Applicability (I)Applicability (I)

Overarching principle– Be practical, focus on a limited number of

features that are most important to the key questions and objectives of the review

Step 1: Report a priori factors affecting the applicability of questions being asked using PICOTS format– Considerations should be reflected in key

questions, inclusion and exclusion criteria for the review


22

Guidance for Assessing Guidance for Assessing Applicability (II)Applicability (II)

Step 1 Actions: – Identify general challenges and specific factors

that may affect applicability Factors chosen will vary based on nature of intervention,

perspective (clinician, policymaker, patient), and outcome (benefit, harm)

– Consult stakeholders, review background Identify factors critical to determining if evidence is

applicable to decisions they need to make Understand current practice to subsequently assess

extent to which studies reflect it

– Extract specific information using PICOTS format


23

Guidance for Assessing Guidance for Assessing Applicability (III)Applicability (III)

Step 2: Review and synthesize the evidence with explicit attention to crucial factors within the PICOTS format

Step 2 Actions:– Identify which of your trials are effectiveness

or efficacy If you have a mix, compare and contrast findings Judge whether differences between the body of

efficacy trials and the real world are important enough to limit its value in making health care decisions


24

Guidance for Assessing Guidance for Assessing Applicability (IV)Applicability (IV)

Step 2 Actions (cont.)– Examine observational studies with more

representative populations to inform judgments about applicability of trial data Population-based studies, pharmacoepidemiologic

studies, registries

– Assess applicability of aggregated evidence Results of effectiveness trials should be highlighted Identify important factors in trials that may impact

applicability and the direction and magnitude of the bias


25

Guidance for Assessing Guidance for Assessing Applicability (V)Applicability (V)

Step 2 Actions (cont.)– Consider subgroup analyses

Seek evidence for empirical relationship between characteristics and effect size Trials done predominantly in males; subgroup

analyses reporting results based on gender can inform the direction and magnitude of the bias

Comparison of event rates across studies can illustrate variation based on population characteristics


26

Guidance for Assessing Guidance for Assessing Applicability (VI)Applicability (VI)

Step 3: Summarize evidence; PICOTS format Step 3 Actions:

– Indicate for each domain (PICOTS) a judgment about whether characteristics of the evidence raise applicability concerns

– Describe not only what study did (exclude patients with history of bleeds) but also the effect it had (low risk of bleeding) and extent this reduced applicability

– Note when major questions of applicability are not addressed and the implications for applicability


27

Summary Applicability TableSummary Applicability Table

Summary Conclusion

Description of Applicability for Evidence Compared to Question

P Describe general characteristics of enrolled patients (proportion with characteristic more helpful than ranges)

Describe how enrolled populations differ from target population and how this may affect benefits or harms

I Describe general characteristics of interventions

Describe how interventions compare to routine use and how this might affect benefits and harms

C Describe comparators used Describe whether comparators reflect best alternative treatment and how this might influence treatment effect


28

Summary Applicability TableSummary Applicability Table

Summary Conclusion

Description of Applicability for Evidence Compared to Question

O Describe what outcomes are most frequently reported

Describe whether measured outcomes reflect most important clinical benefits and harms

T Describe range of followup Describe whether followup used is sufficient to detect clinically important benefits and harms

S Describe settings where studies are conducted

Describe whether settings used may affect applicability


29

Key MessagesKey Messages

Applicability is important and distinct from internal validity

The reviewer needs to evaluate applicability by comparing and contrasting the target population and the study population using the PICOTS format

There are discernable differences between efficacy and effectiveness studies– Effectiveness studies have high applicability

Transparency is an important aspect of the Effective Healthcare Program– A standard approach improves transparency

30

Documents

Systematic Review Module 8: Assessing Applicability C. Michael White, PharmD, FCP, FCCP Professor and Director University of Connecticut/Hartford Hospital