School for Health And Related Research (ScHARR)

Introduction

To

Systematic Reviews

by the Critical Reviews Advisory Group (CRAG)

@ ScHARR

December 1996

1. INTRODUCTION

What is a systematic review?

Why are systematic reviews important?

What is CRAG?

How can CRAG help with systematic reviews?

2. GETTING STARTED

I have an idea for a systematic review. How do I get started?

How do I produce a well-defined research question?

What is a study protocol?

What should I include in my study protocol?

How do I cost a systematic review?

How long is the review likely to take?

3. IDENTIFYING SOURCES

What sources should I consider for the review?

How do I devise a search plan?

Who should do the searching?

What other approaches to searching may be useful?

How do I conduct a manual search?

How do I locate unpublished literature?

4. DEVISING THE SEARCH STRATEGY

What is a search strategy?

What is the aim of my search strategy?

How do I construct my search strategy?

How do I test my search strategy?

5. REVIEWING THE SCOPE AND FILTERING THE LITERATURE

When should I consider modifying my review design?

What tools are available to help me do this?

What else should I take into account?

6. COLLECTING AND EXTRACTING THE DATA

What is my aim when collecting data?

How do I extract data?

What tools are available to help me?

What do I do about missing data?

7. ASSESSING THE QUALITY OF STUDIES

I’ve collected a set of relevant studies for my review. Now what?

Why assess the quality of the studies?

How should I assess study quality?

How do I use the assessment of study quality?

How do I use the results of this process?

8. META-ANALYSIS

What is meta-analysis?

Why do it?

I’m doing a systematic review. Is meta-analysis appropriate?

If yes, how do I do it?

How do I choose the appropriate statistical method?

What help is available?

9. PRESENTATION OF RESULTS

What should be my key objectives in presenting results?

What are my options for presenting results?

10. CONTEXTUAL SUMMARY OF REVIEW

What is the contextual summary?

How do I write my contextual summary?

11. THE WAY FORWARD

I understand what is involved. What now?

Further Reading

ScHARR Introduction to systematic reviews FINAL DRAFT

1. Introduction

What is a systematic review?A systematic review is a “scientific tool which can be used to summarise, appraise, and communicate the results and implications of otherwise unmanageable quantities of research.” (CRD Report 4)1. It is of particular value in bringing together a number of separately conducted studies, sometimes with conflicting findings, and synthesising their results. In this way it can be used for the evaluation of either existing or new technologies and practices. A systematic review is often called an “overview”. David Sackett makes a distinction between the review, the overview and the meta-analysis:-

Review “the general term for all attempts to synthesize the results and conclusions of two or more publications on a given topic”.

Overview “when a review strives to comprehensively identify and track down all the literature on a given topic (also called “systematic literature review”).”

Meta-analysis “when an overview incorporates a specific statistical strategy for assembling the results of several studies into a single estimate”.

Sackett, D et al. Clinical Epidemiology: A basic science for clinical medicine 2nd ed. Little, Brown & Company, 1991.

Although many people use the term meta-analysis interchangeably with systematic review (as in Sackett’s definition, above), strictly speaking it is a statistical methodology used to combine results from different studies into a single summary estimate. This paper observes good practice by using systematic review for the whole process and meta-analysis for the specific statistical technique.

Why are systematic reviews important?There are two main practical reasons for the importance of systematic reviews; first, the limitations of the traditional review and, second, the added power brought by synthesising the results of a number of smaller studies. A “traditional” or “narrative” review may be no more than a subjective assessment by an expert using a select group of materials to support their conclusion. In contrast, the systematic review attempts to be systematic in both the identification and evaluation of materials, objective in its interpretation and reproducible in its conclusions. Smaller studies, such as individual randomised controlled trials (RCTs), may have a rigorous study design but lack the statistical power to demonstrate a statistically significant effect. Antman et al2 have shown that the pooling of results from a number of smaller studies, in a meta-analysis, can lead to earlier and more actionable conclusions both in confirming a beneficial therapy (e.g. anticoagulants in myocardial infarction) and in eliminating one that is not beneficial (e.g. lidocaine) or even harmful.

Politically, systematic reviews are important because the NHS Research and Development Strategy concentrates on the utilisation of research findings rather than focus on the generation of original research which, though useful, may be unable to provide a definitive 1 NHS Centre for Reviews and Dissemination. Undertaking Systematic Reviews of Research on Effectiveness: CRD Guidelines for those carrying out or commissioning reviews (CRD Report No. 4). York: NHS CRD, 1996.2 Antman EM, Lau J, Kupelnick B, Mosteller F & Chalmers TC. A comparison of results of meta-analyses of randomized controlled trials and recommendations of clinical experts. JAMA 1992; 268: 240-248.

or conclusive answer. Internationally, the stimulus for systematic reviews has come from the Cochrane Collaboration, a world-wide group of subject and methodological specialists who aim to identify and synthesise randomised controlled trials in all aspects of health care. Whilst sharing an interest in the synthesis of results, the varied multidisciplinary nature of ScHARR research activities permits a broader perspective both in terms of range of study designs and in incorporating qualitative research.

What is CRAG?The Critical Reviews Advisory Group (CRAG) is an internal ScHARR group, formed in November 1994, with representation from a wide range of disciplines. Its purpose is to advise and provide support to those preparing for, and carrying out, systematic reviews within ScHARR. Subsidiary objectives are to identify or develop resources and methods for use within ScHARR and to support the development and teaching of critical appraisal methods with Sheffield and the Trent Region. CRAG members have produced this introduction to systematic reviews and have identified a range of publications and tools to support both reviews and appraisal. Membership of CRAG is open to any ScHARR staff member with interest or involvement in systematic reviews or critical appraisal. In addition there is a nucleus of representatives from each of the ScHARR sections.

How can CRAG help with systematic reviews?In addition to the tools and resources identified or provided by CRAG, individual CRAG members have agreed to make themselves available to offer advice or support to the systematic review process. The group is willing to meet with potential reviewers to discuss the conduct of a review or to address specific issues arising from the process. e.g. literature searching, meta-analysis, economic evaluations or presentation of results. Although experienced, CRAG members aim to be facilitators rather than experts bringing to bear their knowledge of local skills and resources in support of fellow ScHARR researchers.

The following sections will guide a researcher through the systematic review process using a simple question and answer format.

SCHARR Introduction to systematic reviews FINAL DRAFT

2. Getting started

I have an idea for a systematic review. How do I get started? Once you have a rough idea, you need to talk the topic through with both subject and methodology experts. Such contacts, along with a preliminary survey of the literature, should ensure that your idea is both worthwhile and achievable. You will need to survey the literature to ensure that the topic is necessary (in terms of existing reviews) and feasible (in terms of the volume of literature in the field).

How do I produce a well-defined research question?You may wish to convene a “scoping meeting” to discuss the research idea, and from this, produce a well-defined research question. You should define the suggested systematic review in terms of:

· population and medical/social problem to be investigated;

· intervention and comparator under scrutiny;

· outcomes of concern;

· design of study to be reviewed, e.g. randomised controlled trials.

Once you have formulated the research question you should work it up into a study protocol and proposal.

What is a study protocol?Your study protocol should set out the tasks and methods to be followed in the systematic review. You will need clear and explicit methods to minimise bias and to ensure that results are reproducible.

What should I include in my study protocol?There is no single protocol format, nor is there one accepted set of contents that needs to be followed. The Cochrane Collaboration have a fairly prescriptive format that you may wish to adapt to your specific needs. The structure is very similar to that of this Introduction to Systematic Reviews.

How do I cost a systematic review?You should discuss your review with Information Resources staff who will help you estimate the costs of staff time, inter-library loans, photocopying, postage, etc. They can also provide a template to help in the costing of systematic reviews. Different topics, scopes and styles of review influence costs. For example, a very meticulous search for unpublished data necessitates more expenditure on postage.

How long is the review likely to take?You will need to set the systematic review process within a realistic timeframe. Typically, a review will last between 6 and 18 months and it will cost somewhere between £ 20,000 and £ 75,000.

How does all this fit together?The scope of the review as described in the protocol, together with its associated costs and timeframe, should produce a coherent study plan. As with the protocol, no single framework is possible, however a typical plan/application will cover;

· objectives;

· rationale;

· design of the review;

· time-chart for major activities, and

· budget.

·

SCHARR Introduction to systematic reviews FINAL DRAFT

3. Identifying sources

What sources should I consider for the review?Your systematic review should attempt to be comprehensive in its coverage of bibliographic sources. Broadly speaking the choice of sources is determined by such factors as the discipline(s) of a particular study, the types of materials to be included, the period of time to be covered and the availability of previous bibliographies or reviews. In deciding the sources to be used you need to consider the inclusion of general medical, health and nursing databases(e.g. MEDLINE and CINAHL), specialist subject databases (e.g. Cancerlit), databases from other disciplines (e.g. ERIC for education or PsycLit for psychiatry), databases of reviews (e.g. DARE and the Cochrane Library) and databases for specific types of materials (e.g. Dissertation Abstracts or the Index of Scientific and Technical Proceedings). It is helpful if you can define the disciplines and types of material you require and then work with an information specialist in identifying the relevant sources for these.

How do I devise a search plan?Your search plan will be a brief summary of the topic of the study, a summary of inclusion and exclusion criteria, a list of the appropriate publications, sources and methods that you will use for identifying materials and a “brainstorm” of associated or concepts related to your topic. You will find it helpful to work with Information Resources staff in producing your search plan.

Who should do the searching?You should view the search process as a partnership with the researcher possessing the subject and/or methodological expertise and the information specialist being able to formulate enquiries and interrogate databases. Initially you may wish to sit alongside the information specialist during a search of a primary database to assess and refine the strategy based on materials that are retrieved. Subsequently, with increasing confidence and explicit instructions, the searcher is able to interpret this strategy into the specific terminology of each additional database. You will need regular communication and feedback.

What other approaches to searching may be useful?In addition to subject searching you may be asked to identify key articles that are of importance to your topic. You will be able to use citation databases to search for any paper that cites a seminal article. You may also identify authors or institutions working in your particular topic area and search for papers originating from these sources. Other approaches involve checking all the references at the end of relevant articles or manually searching through printed indexes, bibliographies or issues of relevant journals.

How do I conduct a manual search?You will usually restrict manual searching to key journals in a particular topic area. You will thus overcome deficiencies in indexing or in database coverage. The Baltimore Cochrane Centre has produced guidelines on manually searching journal issues. Before embarking on a manual search you would do well to check whether any of the Cochrane Groups have already searched your key journals. Your review of key journals can be done at the level of the contents page, the abstract or the whole article depending on resource constraints but must be systematically conducted and documented.

How do I locate unpublished literature?You may complement your literature search by contacting subject experts or centres of excellence. You can do this through written correspondence or through e-mail discussion lists. You can use printed directories or the “Institution” field of databases to identify potential contacts. Increasingly, you will find it useful to search the Internet for departmental profiles and research interests or for registers of trials in progress.

ScHARR Introduction to systematic reviews FINAL DRAFT

4. Devising the search strategy

What is a search strategy?Whereas the search plan records the sources and concepts for your review the search strategy operationalises these decisions for an individual database. Some of your decisions will be easy to implement (e.g. inclusion/exclusion of languages or types of study) but others will require an optimal mix of subject headings (assigned by the indexer) and free-text terms (assigned by the author e.g. title or abstract). You may find that subject headings prove inappropriate or too specific for your topic whereas free-text terms may have been used inconsistently or imprecisely. You will need a combination of both approaches and will need to rehearse these for each additional database.

What is the aim of my search strategy?Your search strategy will aim to minimise non-retrieval of relevant documents and retrieval of irrelevant documents and to maximise retrieval of relevant documents. The eventual strategy should more or less correspond to the intended coverage of your review. Although you will need to aim for a satisfactory compromise between quantity and quality, the requirement for the search to be systematic means that you should err on the side of retrieval of too many items. You will find that having subject experts on your team judge the relevance of items once retrieved is more reliable than entrusting the review entirely to the vagaries of indexer subjectivity and inconsistency.

How do I construct my search strategy?Your search strategy will also include syntax that expresses the relationships between search terms. Most databases use what is known as “Boolean logic”. Simply explained, synonyms or associated concepts are joined together with “OR” (e.g. nurse OR health visitor OR midwife) whereas intersecting concepts are joined together with “AND” (e.g. Infant-Newborn AND mass-screening). An information specialist can advise on these issues and validate the internal logic of your search statements.

How do I test my search strategy?Researchers often underestimate the time taken to develop an optimal search strategy. You must judge the relevance of retrieved items to your topic. Can you identify any aspects that seem to be missing? Do any papers suggest additional concepts or search terms? Check items that you already know of. Have they been retrieved? If not, why not? Revision of the search strategy is thus an iterative process.

ScHARR Introduction to systematic reviews FINAL DRAFT

5. Reviewing the scope and filtering the literature

When should I consider modifying my review design?Subject experts are often surprised by the large quantity of material in a specific topic area. A primary database may contain no more than 35% of total items relevant to your topic. It is not unusual, therefore, for you to have to revise your entry criteria in the light of initial results. Certain approaches are more legitimate than others. The first area for you to review is your definition of the research question; have you defined the dimensions of population, intervention and outcome tightly enough? Is your review to look only at comparable populations or settings? Is it to look at a specific intervention rather than a process of care? Is there a broad base of knowledge around measurement of particular outcomes? Secondly, might you limit inclusion of materials according to specific types of study? e.g. randomised controlled trials or clinical trials. Finally, are your criteria for language or year restrictions realistic? Such explicit and systematic criteria are preferable to arbitrary decisions about the number of sources searched, the number of articles retrieved or the thoroughness of a search strategy.

Less commonly retrieval might yield fewer items than expected. Once you have verified your search strategy by checking for “known items” you may extend the number of sources searched, drop a methodology restriction or pursue unpublished or non-English materials more vigorously. However, a rigorous review of a few high-quality studies is considered preferable to a less discriminate trawl for items of questionable worth.

What tools are available to help me do this?Information specialists at McMaster University, the Cochrane Collaboration and the NHS Centre for Reviews and Dissemination have devised optimal search strategies to discriminate for appropriate study designs e.g. review articles, randomised controlled trials or articles on diagnosis, aetiology, prognosis and therapy. You will find these and other filters (for studies such as guidelines and economic evaluations) available in the ScHARR Library.

What else should I take into account? As with other types of research, you will need to exercise caution if you revise the remit of your systematic review. Important considerations are the nature of your original agreement with the funding body, the relevance of the revised study to the originating clinical or purchasing “problem” and the financial implications of such decisions (i.e. additional expenditure on literature searches and photocopies or provision for payback of surplus funds).

ScHARR Introduction to systematic reviews FINAL DRAFT

6. Collecting and extracting the data

What is my aim when collecting data?Your aim is to provide sufficient data to make reliable assessments at each step of the review process. Your initial assessments are based on the relevance of each article to the study question and can be carried out from minimal information such as the title, abstract and keywords. You should “include in” any items with insufficient details to prevent premature exclusion of relevant materials. You will need to obtain photocopies of all relevant or potentially relevant items. You will then assess these for relevance, completeness of information and methodological quality. You should then code and record all your decisions, either on a database or a separate data extraction form. You will find such forms a useful summary for easier handling of large numbers of articles or variables.

How do I extract data?You can take data such as the bibliographic details (authors, titles and source) directly from each article. You can often download these, together with an abstract, from a bibliographic database, such as Medline, and then import them into personal reference management software such as Reference Manager. User defined fields, as provided by Reference Manager, can be coded for the presence of different types of information e.g. costs, prevalence, outcomes, etceteras or for specific outcomes. Alternatively you can include this information on a data extraction form. Examples of data extraction forms are given in Appendix 3 of the NHS Centre for Reviews and Dissemination Report No. 4

What tools are available to help me?In addition to Reference Manager, which can be used to handle bibliographic and document management functions, you have a number of options for managing the review process. RevMan, the Cochrane Collaboration’s review management software, can be used to handle the results of individual studies and to integrate and synthesise the variables. Alternatively multipurpose databases such as SPSS or Access may be used to present cumulative data across a number of studies.

What do I do about missing data?If all the information you require is not available from an article you will have to try alternative sources. Many articles are summaries of much larger reports of studies. These are much more likely to have the information needed. Certainly, if the study was funded by a research council grant (eg. MRC, NHS, ESRC etcera) the funding organisation should have a copy of the report. Many organisations have data archives where fuller project details are deposited, indexed and archived. The extent of your efforts to pursue missing data depends on the nature and extent of missing variables. Generally, efforts to obtain items of data for articles that have met all other quality control criteria are more worthwhile than for articles that are of marginal benefit. Authors may still be contactable at the institutional address given in the article. Prior to this it will be worth your conducting follow-up author searches to try to identify associated published or semi-published studies. You will find that this is particularly the case with economic data given the tendency for authors to report results in separately published, or unpublished, studies.

ScHARR Introduction to systematic reviews FINAL DRAFT

7. Assessing the quality of studies

I’ve collected a set of relevant studies for my review. Now what?Once you have obtained the papers relevant to your review, the next task is, systematically, to assess the quality of the studies. This is the point at which you actually have to read the studies!

Why assess the quality of the studies?Different studies may each examine the same issue in different ways, or indeed examine different, but related issues. The quality and strength of evidence different studies provide in answer to your own particular research question is likely to vary widely among the papers you have collected. Naturally you want to give greater emphasis to the “better” studies in reaching your conclusions.

But what do we mean by “better”?

· Relevance to the research question: even after all the scoping, filtering and limiting, some studies turn out to be of limited applicability to your question. You may have to read a paper quite carefully to decide on the generalisability (or external validity) of the study.

· Validity of the study conclusions: even in studies of apparently robust design the results or conclusions may have been afflicted by the effects of chance, bias or confounding. Methodological flaws limit the degree to which the study conclusions are warranted by the data (that is, the internal validity).

How should I assess study quality?In a systematic review, the process for assessing study quality should aim for the same high standards as every other step of the review. Whatever the process, it should as far as possible be explicit, documented, repeatable, valid and free from bias — just like the studies themselves.

Ideally, those assessing study quality should not be influenced by extraneous material, such as the names or institutional affiliations of the authors. You may want assessors to examine the studies “blind” — that is, to see only the methods and results section from each paper.

You may find it helpful to use one of the many published “critical appraisal checklists” to help you assess the studies. In using checklists, you should bear a number of points in mind:

· different checklists exist for different disciplines and for different study designs: you may need to get advice on the most appropriate checklist to use

· some checklists simply remind you what to look for; others attempt to assign a score or grade to a study

· checklists are only an aid to good critical appraisal, not a method in themselves; even the best checklist does not relieve you of the need to make informed and thoughtful judgements

How do I use the assessment of study quality?First, you can use the assessment to exclude from the review studies which fall below your standard of relevance or validity. Second, for studies to be included, you use the assessment to rank or group studies by quality, so that you can give greater emphasis to the “best” evidence available. Depending on your approach to the critical appraisal process, you may be able to rank, grade or score the studies which you include in the review.

How do I use the results of this process?Finally, having organised the studies by quality and relevance, you may adopt either an interpretative or a statistical approach (as discussed in the next section) to combine your results. In either case, you have the option of giving greater weight to the results of better studies.

ScHARR Introduction to systematic reviews FINAL DRAFT

8. Meta-analysis

What is meta-analysis?By meta-analysis we mean “the statistical analysis of the data from a collection of studies in order to synthesise the results”.We use the term to describe the statistical technique but, as previously mentioned, others may describe a systematic review in which meta-analysis has been used as “a meta-analysis”.

Why do it?The rationale for meta-analysis given by Mulrow3 is “to increase the power and precision of estimates of treatment effects and exposure risks”.Thus, you may want to add a meta-analysis to a systematic review to obtain the best overall estimate of the effect of an intervention or treatment. By combining the results of all the available studies you increase the power of your study to detect important treatment effects. It is important to remember that while a systematic review and meta-analysis can be expensive, it is cheaper than performing a large new trial.

I’m doing a systematic review. Is meta-analysis appropriate?In answering this, you will need to consider two further questions.1. Can the results of the studies be expressed as an “effect measure”, (such as an odds

ratio or rate difference) which has a numerical value and can be combined into a single estimate?

2. More importantly, does it make sense to combine the results of the different studies into a single estimate?

You should only answer “Yes” to question 2 if the interventions being compared, the populations receiving the interventions and the outcomes being measured are sufficiently similar across all studies.

If yes, how do I do it?Meta-analysis is the final, and not necessarily the most important, stage of the process outlined below. You will be able to obtain many of its benefits simply by graphically representing the results of different studies.After systematically identifying relevant studies, you will need to choose the most appropriate effect measure for combining them. You should then assess the quality of the studies and where possible rank or grade them. You will then represent the results of the individual studies on a graph which shows a point estimate of each study’s effect size, with its confidence interval indicated by a horizontal line on either side of the point. An example of this type of graph is shown in Figure 1:

3 Mulrow CD. Rationale for systematic reviews. In: Chalmers I, Altman DG (eds) Systematic Reviews. BMJ Publishing group 1995.

Figure 1. Meta-analysis of 33 trials of streptokinase for myocardial infarction (from Mulrow).

You can place studies on the graph in chronological order (as above) but can also rank them in other ways E.g. by quality, to show visually whether the results of better quality studies are similar to those of other studies of lesser quality.At this point you can decide whether or not to proceed to a formal meta-analysis (statistical synthesis). The graph will help you to decide whether the results of the individual studies are sufficiently similar for you to combine them into a single estimate. If you decide to synthesise the results you will need to choose an appropriate method of statistical synthesis.

How do I choose the appropriate statistical method?Your two key questions are:1. which outcome measures shall I combine?2. which “model” should I use? There are two main models that are used for meta-analysis: the fixed effects and the random effects models. There is considerable debate about which of these should be used. A decision is based partly on the data and partly on philosophy. You will need to seek advice from one of the ScHARR statisticians.

What help is available?ScHARR has a range of resources (people, books, articles and software) to help you undertake a meta-analysis. You will find a collection of articles and books on meta-analysis in the Library.

ScHARR Introduction to systematic reviews FINAL DRAFT

9. Presentation of results

What should be my key objectives in presenting results?In presenting the results of a systematic review, you should aim to give the reader a clear idea of each of the following:

· the major conclusions of the review in answering the initial research question(s)

· the sources of evidence on which those conclusions are based

· the quality of the evidence supporting each conclusion

· the likely degree of completeness of the search for evidence

Optionally, the results of a review may also include:

· the results of meta-analysis

· reviewed evidence in computer-searchable format

What are my options for presenting results?Because systematic reviews are frequently based on the critical appraisal of large numbers of studies of differing design, quality and conclusions, you will want to use a mixture of textual commentary and summary tables to present the search strategy, the evidence identified and the conclusions in as clear a form as possible.

Textual commentarySimple narrative is the commonest way to present findings. It is almost always the case, even for reviews which have identified very limited numbers of studies, that a narrative is enhanced by including tables (see below).

You should always structure your textual commentaries, usually by appropriate headings and subheadings, to indicate

· the research question currently under consideration

· the types or quality of studies being reviewed

Summary tablesYou may want to use tables to present a range of aspects of the review process and results in easily understood summary form. For example, you might present:

· the search strategies and resulting yield of studies in a table showing databases used, search terms used, total publications found, and relevant publications found;

· the studies contributing to the review in terms of their methodological features in a table showing study citation, country and year, study design, study size, outcome measures, overall grading of study quality;

· the studies contributing to the review in terms of their conclusions in a table showing study citation, outcome measures, effect size, confidence intervals, strength of evidence for, or against, intervention.

Meta-analysisYou should always present the results of meta-analysis as a diagram or table showing outcome measures and confidence intervals for each individual study included, for all studies, or for various subsets of studies taken together.

ScHARR Introduction to systematic reviews FINAL DRAFT

10. Contextual summary of review

What is the contextual summary?Your contextual summary will follow your report of your methods and analyses. You will interpret and discuss your results and set out the implications of your review for practice and research.

How does this fit in with the overall structure of the report?You may wish to follow the following exemplar report structure where the contextual summary comprises the “Discussion” and “Implications of Review”:

· Abstract or Executive Summary· Background Information· Hypotheses Tested in the Review· Review Methods· Details of Studies Included in the Review· Details of Studies excluded from the Review· Results of the Review· Analysis of the Robustness of the Results· Discussion· Implications of the Review· Reference Lists· Dissemination and Further Research

How do I write my contextual summary?

What is a “logical” structure for my contextual summary?Your summary will fall into two main parts:

· Discussion and interpretation, where you analyse the strengths and weaknesses of your results.

· Implications of your results (for the purchase and provision of health care, and further research).

What should I discuss in the contextual summary?You should summarise the results of the study with respect to the primary research question, and then:

· qualify these results by noting the possible impact of biases in the primary data, or through missing data, etc.

· interpret your results more widely using sub-group analyses and other evidence from outside the review. You should only use external literature to produce a short interpretation of those findings that you cannot adequately explain using only evidence identified within the systematic review.

You need to make this distinction to separate evidence that has been identified systematically and critically reviewed from that which has not and, as a consequence, is more open to bias.

How should I handle the implications of my review?Your review should address recommendations for both future health care and future research. You should use wider interpretation based on external “evidence” to identify areas for future research, not health care practice.

11. The Way Forward

I understand what is involved. What now?This introduction is just an orientation guide. Other general publications that you will find useful are the Cochrane Collaboration HandbookI and the Centre for Reviews and Dissemination Report No. 4II together with the BMJ publication, Systematic ReviewsIII. The NHS Centre for Reviews and Dissemination has a useful set of World Wide Web pages on systematic reviews. The Cochrane LibraryIV on CD-ROM is home to the Cochrane Collaboration Handbook, all the Cochrane Reviews and details of other reviews and articles about doing reviews. ScHARR produces a bibliographic guide; The ScHARR Guide to Evidence Based PracticeV, and maintains a Reference Manager database on the methodology of reviews together with topic files on all aspects of the review process. Ask at the ScHARR Library for details.

You may also want to talk to some of the members of CRAG. Enquire at the ScHARR Library for details of the current membership of the Group.

Further Reading

I Oxman AD (ed). Section VI: Preparing and Maintaining Systematic Reviews: The Cohrane Collaboration Handbook. Oxford: Cochrane Collaboration, 1994.II NHS Centre for Reviews and Dissemination. Undertaking Systematic Reviews of Research on Effectiveness: CRD Guidelines for those carrying out or commissioning reviews (CRD Report No. 4). York: NHS CRD, 1996.III Chalmers I and Altman DG. Systematic reviews. London: BMJ Publishing Group, 1995.IV The Cochrane Library [database on disk and CD ROM]. Oxford: The Cochrane Collaboration & Update Software, 1996.V Booth A. SCHARR Guide to Evidence Based Practice. (SCHARR Occasional Paper No.2). Sheffield: SCHARR, 1996.