1

“Ensuring relevance and building enthusiasm for Cochrane reviews: determining appropriate methods for identifying priority topics for future Cochrane reviews”

Priority-Setting Workshop

Cochrane Colloquium Freiburg, Germany

Saturday, October 4th, 2008 CHAIRS: Peter Tugwell, Elizabeth Waters PRESENTORS: Mona Nasser, Jodie Doyle, Vivian Welch. FACILITATORS: Jodie Doyle, Mona Nasser, Peter Tugwell, Erin Ueffing, Elizabeth Waters, Vivian Welch Objectives:

1. To inform the participants about different existing priority-setting exercises and also to introduce a framework for evaluating these exercises.

2. To illustrate the current priority-setting exercises undertaken by different Cochrane entities and their relationship to new funding for reviews and thus new reviews

3. To discuss how these different priority-setting exercises could be used for prioritizing topics for Cochrane reviews and to discuss the relevant benefits and limitations.

Background: In April 2005, a joint meeting of Cochrane Centre Directors, Coordinating Editors and Steering Group members, it has been recognized that a strategic view needs to be taken to improve the priorisation process for topics of Cochrane reviews to ensure the relevancy of the reviews to national and international stakeholders. Consequently, the Cochrane prioritization fund has been created, this resulted that five projects (http://www.cochrane.org/admin/cc_funding_initiatives.htm#oppfund) has been funded by the Cochrane Collaboration using different methods to prioritize topics for Cochrane reviews. This included a joint project of the Cochrane Health Equity field (Peter Tugwell, Vivian Welch, Erin Morris), Cochrane Public Health Review Group (Elizabeth Waters, Jodie Doyle), Cochrane Developing Countries Network (CDCN) (Mona Nasser, Zulma Ortiz (previous coordinators of the CDCN)) and the Oslo satellite of Effective Practice and Organization Care (EPOC) Cochrane

 

  2

Review Group (Andy Oxman) entitled “Prioritizing Cochrane review Topics Relevant To Low- and Middle Income Countries (LMIC)”. Presentations:

1. Evaluating the priority setting processed used across the Cochrane Collaboration: Accountability, Reasonability and Equity (Mona)

2. Priority setting for Public Health reviews - an example and lessons learned (Jodie)

3. Strategies to improve priority-setting to consider health equity in Cochrane entities

(Vivian) Discussion: The participants were divided with four groups and each of the groups has discussed one or two of the following discussion questions and provided a feedback at the end:

1. How to select and rank-order criteria for prioritisation? 2. Who should be consulted, and how, in setting and applying criteria? 3. Evidence mapping-how to judge applicability, extrapolation, plausibility of existing,

available systematic reviews? 4. How the prioritisation strategies could be integrated into the current roles of all entities

within the Collaboration; 5. How to align priority setting with new strategic funding opportunities

Group 1 (Facilitators: Erin and Liz): (How to select and rank-order criteria for prioritisation?) - Involve all stakeholders; will be more cost-efficient since criteria do not need to be duplicated for different stakeholders - Open and transparent process, need to describe what, why and how - Critical to ask the right questions. Again, an issue of resources: questions that are unclear or misinformed cannot be answered effectively. - Rejected the idea of rank-ordering criteria; criteria need to be selected in context and need to be flexible Group 2 (Facilitator: Mona): (Who should be consulted, and how, in setting and applying criteria?) The participants have named different individuals that should be involved in the prioritisation process, this included consumers, clinicians, researches, funders and etc. It was also discussed that it might be necessary to involve individuals from different regions and settings (e.g. all of the regions of the country or both developing and developed countries). There was no consensus whether all of them should be involved during the whole process including the setting or

 

  3

applying criteria or not but the participants agreed that this depends on the objective of the prioritisation process and there are different methods to involve them through mapping workshop, existing networks, focus groups, online surveys (snowball sampling). One of the members from the global mapping project in Australia mentioned that they found online surveys an easy method to reach a wide group of individuals including consumers, however, it is difficult to determine whether all stakeholders were equally involved and understand whether there is a different between those who responded or did not respond. There were several points raised that needs to be considered during the process:

1. In a discussion on whether consumers and other stakeholders should be both involved in setting and then applying criteria, it has been mentioned if the consumers or other stakeholders would not be involved in setting criteria, they should have sufficient education and training to understand the criteria.

2. If the participants would not feel ‘ownership’ to the prioritisation project, they might not be actively involved in it. This could be also an important factor to keep people involved in the process to appeal or challenge the results of the prioritisation project.

3. The project investigators needs to decide in an early stage of the process whether how broad or narrow the prioritisation process would be.

Group 3 (Facilitator: Jodie, summarized and presented by Tracey Perez Koehlmoos ): (Evidence mapping-how to judge applicability, extrapolation, plausibility of existing, available systematic reviews?) It is useful and important to identify the gaps of evidence (whether there is a lack of systematic reviews), however, to explore the applicability and extrapolation, we need to evaluate whether the evidence from the systematic review is also relevant to different settings e.g. resource poor settingsfor example in the SUPPORT summaries, each author makes a judgment about whether results can be extrapolated to resource-poor settings, but often they lack any data to make this judgment (e.g. if all studies are conducted in high-income countries) Group 4 (Facilitator: Peter and Vivian, summarized and presented by Kent Ranson) (How the prioritisation strategies could be integrated into the current roles of all entities within the Collaboration; how to align priority setting with new strategic funding opportunities?) -Priority-setting for systematic reviews needs to be demand driven -need to also drive primary research (e.g. if systematic reviews do not exist or are inconclusive, leading to treatment uncertainty) -Need to test methods in Cochrane review Grous (CRGs) -Who is the audience- clinician, consumer, funders? -How do you involve multiple stakeholder groups (e.g. very few examples of clinicians and patients working together- e.g. James Lind Alliance) -Lack of awareness of priority-setting methods by Cochrane review groups -Avoid overlap between CRGs or work together to set priorities where there are areas of mutual interest

 

  4

-Priority setting may identify complex questions that are not suited to beginner reviewers, and thus justify the need for funding of an experienced team -Need to get funding agencies on side to support high-priority systematic reviews- but these do not need to be large funders (e.g. National Health Service, Canadian Institutes of Health Research), they can also be small funders. Useful Links: Cochrane Health Equity Field (www.equity.cochrane.org) Cochrane Public Health Review Group (www.ph.cochrane.org) Evidence Aid (http://www.cochrane.org/evidenceaid/index.htm) SUPPORT Collaboration (http://www.support-collaboration.org/index.htm)

Evaluating the priority setting processes used across the Cochrane Collaboration:

Accountability, Reasonability and Equity

Workshop title: Ensuring relevance and building enthusiasm for Cochrane reviews: determining appropriate methods for identifying priority topics for future Cochrane reviews

Cochrane Colloquium, 3-7 October 2008Freiburg, Germany

There is no simple way to set priorities. However, failure to establish a process for priority setting has led to a situation in which only about 10% of health research funds from public and private sources are devoted to 90% of the world’s health problems (measured in DALYs). This extreme imbalance in research funding has a heavy economic and social cost. To make matters worse, even the 10% of funds allocated to the 90% of the world’s health problems are not used as effectively as possible, as health problems are often not prioritized using a defined methodology.

Global Forum for Health Research

Prioritizing Cochrane Review Topics Relevant to Low- and Middle-Income Countries

Peter Tugwell, Vivian Welch, Erin Ueffing, Zulma Ortiz, Mona Nasser, Elizabeth Waters, Jodie Doyle, Andy Oxman

1) Identify what priority-setting approaches are underway by Cochrane entities already and whether/how these can inform priorities for systematic reviews of relevance to LMIC

2) Explore different methods of identifying priorities for systematic reviews for specific audiences, recognizing that priorities may be different (eg for policy-makers, for practitioners). We might convene small working groups of practitioners to identify priorities for systematic reviews at a national or regional level

3) Map existing systematic reviews to identified priorities to identify gaps/needs for new systematic reviews, as well as showing whether the Cochrane Database of Systematic Reviews already answers high-priority questions relevant to LMIC

Background:

• Ebrahim S, Moore T. Priority setting for review topics in the Cochrane Review Groups. 7th Cochrane Colloquium, 5-9 October 1999, Rome, Italy

• Ghersi D, Kennedy G, Rio P, Shea B. Consumer Setting Priorities for Cochrane Review Groups. 5-9 October 1888, Rome, Italy

• Vet HCW, Korese MEAL, Scholten RJPM. The efficacy Of treatments for chronic benignpain disorders: setting research priorities by literature searches With minimal reading. 8th Cochrane Colloquium 25-29 October 2000, Cape town, South Africa.

• Survey of prioritisation, commissioning and cochranisation of non-Cochrane reviews, Adrian M Grant, CCSG, Khon Kaen, April 2006

• Bellorini J, Doree C. A method for priortising review topics: map of distribution of randomized controlled trials (RCTs) in the Cochrane Review Group Trials Register. 14th Cochrane Colloqiuium 23-26 October 2006, Dublin, Ireland.

• Doyle J, McDonald L, Bailie R, Waters E, Armstrong R. Prioritising and promoting topics for systematic reviews to address the public health needs of indigenous people. 14th Cochrane Colloqiuium 23-26 October 2006, Dublin, Ireland.

• Nasser M, Lodge M, Fedorowicz Z. The relevance of Cochrane Reviews to the Cancer Priorities in Iran. 15th Cochrane Colloquium 23-27 October 2007, Sao Paulo, Brazil.

• We have contacted 67 Cochrane review groups and Cochrane fields.

• 52 groups responded (about 78%)

• 17 did not have a priority setting exercise(about 25%)

• 8 had some relevant editorial process for accepting titles (about12%)

• 27 had a prioritsation process or planned one (about 40%)

Summary of ways the CRGs make decisions on prioritising new or updated reviews:

1. Adapting priorities from other external organizations.

2. Using the database of uncertainties (DUETS)

3. Asking for recommendations and suggestions from editors (email, survey, face-to-face meetings), authors or other members of the Cochrane Entities (CEs) (survey, email, web based suggestion box). This might be accompanied by a further step on ranking the titles in an editorial board discussion or emailing members of the CEs.

4. Identifying gaps in the existing literature (topics on which a number of trials have been undertaken but no Cochrane review or systematic review exists) in areas which are relevant (as judged by editors, members of the CEs, other stakeholders or identified in guidelines).

5.Using a consensus process involving different stakeholders in the form of a workshop, Delphi process, survey or focus group

6.Using health indicators e.g. Mortality or incidence to prioritise reviews.

Criteria for priority setting used by CEsClinical relevancy and importanceKnowledge of new trials available or numbers of them availableImproving patient outcomesImportance to NHS priorities (UK based CRG)Achievability and resources requiredImpact on efficacy and efficiencyTimelinessRelevancy to the country (Australian based CRG)Focus of the CRGInterset and priority as defined by individual membersHow might this Cochrane review contribute to achieving the United Nationa‘sMillenium Development Goals in infectious diseases, maternal motrality orchild health? (Infectious diseases CRG)Burden of Disease, magnitude of problem and urgencyImportance to developing countriesLarge scale impact on population.Opportunity for action

Challenges:

1. Volunteer based contribution of the author’s

2. The priority setting could be useless if authors would not be interested to take over any of the topics (Some fields provide bursary schemes in the specific topic)

3. Inexperienced authors can not necessarily undertake a prioritize topics which may encounter methodological difficulties

4. Lack of funding and time

How to evaluate or compare the processes?Some examples:

• “Accountability for Reasonableness” (A4R)(1, 3)

• Comparison of various priority setting approaches in the Combined Approach of Matrix (2)

• Programme budgeting and marginal analysis (PBMA) (3)

References:1. Martin D, Singer P. A Strategy to improve priority setting in health care institutions. Health care analysis 2003, 11 (1): 59-68.2. Ghaffar A, Francisco A, Matlin S. The Combined Approach Matrix: A priority setting for health research. June 2004.3. Gibson J, Mitton C, Martin D, Donaldsen C, Singer P. Ethics and economics: does programme budgeting and marginal analysis contribute to fair priority setting? Journal of Health Services Research & policy 2006: 11(1) : 32-37.

A4R is an ethical approach to priority setting that seeks to ensure fairness in how priority-setting decisions are made.

Relevance: Decisions should be made on the basis of reasons (i.e. evidence,principles, values, arguments) that ‘fair-minded’ stakeholderscan agree are relevant under the circumstances.

Publicity: Decisions and their rationales should be made available to stakeholders.

Revision and appeals: There should be opportunities to revisit and revise decisions in light of further evidence or arguments, and thereshould be a mechanism for challenge and dispute resolution.

Enforcement: There is a voluntary or regulatory mechanism for ensuring that the other three conditions are met

“Accountability for Reasonableness” (A4R)

A possible framework to compare the strategies:

• Objective of priority Setting and process• Inclusiveness• Equity• Evaluation of the existing evidence coverage • Criteria for priority setting• Continuity (expect of the normal procedure of a CRG) and

implementation• Relevance (A4R criteria)• Publicity (A4R criteria)• Appeals (A4R criteria)• Enforcement(A4R criteria)

14 of the processes used by Cochrane entities could be categorized in this table.

Discussion Questions:

How to select and rank-order criteria for prioritisation?

Who should be consulted, and how, in setting and applying criteria?

Evidence mapping-how to judge applicability, extrapolation, plausibility of existing, available systematic reviews?

How the prioritisation strategies could be integrated into the current roles of all entities within the Collaboration;

How to align priority setting with new strategic funding opportunities

mona.nasser@iqwig.de

Developing HPPH review priorities (2002 project)

• Development and use of a taskforce of individuals to identify and nominate global research needs

• Identification of gaps in the current systematic review literature

• Production of a list of useful review toipics for decision making within public health agencies globally

• Prioritisation of the nominated review topics, using a priori set of criteria

• Communication and dissemination of the list of prioritised review topics

Global Priorities Taskforce

• David McQueen: Assoc. Dir, Global Health Promotion, Nat Center for Chronic Disease Prevention and Health Promotion, CDC, and IUHPE

• Priscilla Reddy: Dir, Health Promotion Research and Development Group, MRC of South Afirca

• Derek Yach: Exec Dir, Non-Comm Diseases & Mental Health, WHO• Gauden Galea: Regfinal Adviser, WHO, Western Pacific, Manila• Metin Gulmezoglu & Annie Portela, Dept of Reproductive Health and

Research, WHO• Andres de Francisco, Head, Scientifis Operations, Global Forum for

Health Research• Peter Briss: Acting Chief, Community Guide Section, CDC Guide to

Community Preventive Services• Tony Stewart: International Health Unit, MacFarlane Burnet Institute• Jodie Doyle and Elizabeth Waters, Cochrane HPPF Field

Scoping HPPH reviewsSept 2002:• Cochrane Library• Effective Public Health Project (Canada)• The Guide to Community Preventive Services• HAD Effectiveness Reviews (UK)• Eppi Centre reviews• NHS R&D HTA Programme

Results381 systematic reviews within the Field’s scope186 were Cochrane reviews or protocolsMostly infectious diseases (of which 80% were on vaccine effectiveness), tobacco control, and sexual health (including HIV prevention)

• Criteria for priority setting – burden of disease(1) Burden of disease, magnitude of problem, urgency; (2) Importance to developing countries; (3) Large scale impact on population; (4) Opportunity for action

Prioritised topics

• Not directly aligned to specific diseases – eg. Interventions that aim to reduce health risk behaviours through enhancing protective environments for adolescents

• Many were multi-faceted – eg. Community-building interventions to improve physical, social and mental health

• Mental health promotion topics – eg. Pre-natal and early infancy psychosocial interventions for preventing mental disorders

• Nutrition topics – eg. Interventions focusing on adolescent girls in order to improve nutritional status of women of child-bearing age prior to first pregnancy

Implementation strategies

• Dissemination of prioritised topic- Field website- Cochrane Review Groups- Utilise Taskforce members’ organisations- Approach funding sources

• Identification of reviewers• Consultation with potential users ie policy

makers and practitioners, re-content (interventions, outcomes)

• Promotion of review findings to potential users

Lessons

• Scoping exercise crucial to determine first what’s already out there

• Engagement with stakeholders invaluable in creating closer ties/creating advocates

• Need ability to see through editorial process to be of most value

Improving priority-setting to consider health equity in Cochrane entities

Cochrane Colloquium 2008

www.equity.cochrane.org

"The term 'inequity' has a moral and ethical dimension. It refers

to differences which are unnecessary and avoidable but, in

addition, are also considered unfair and unjust.“

- Whitehead, 1991

www.equity.cochrane.org

Equity Evidence Aid

Does it work in the poor?Does it work in the disadvantaged?

Copyright WHO/P. Virot

www.equity.cochrane.org

Equity Evidence Aid

1. What can be applied/extrapolated from existing systematic reviews to disadvantaged populations?

2. Priority-setting to fill gapsFair process (eg AFR)Criteria to consider equity

www.equity.cochrane.org

Equity Evidence Aid Process

Mapping evidence

Describe and Evaluate priority setting

Equity Evidence Aid

Improve priority setting for health equity

Priorities Database

www.equity.cochrane.org

Other priority-setting approaches

James Lind AllianceTop 10 biotechnologies to improve health in LMIC (Daar et al)FINOHTA

www.equity.cochrane.org

DankeschDankeschöönnvivian.welch@uottawa.cavivian.welch@uottawa.ca

Ottawa