Level V Evidence

From HoHouston, Texnesota; UConoratory, UCoHershey Bon

The authofunding: J.DArthroscopSynthes, andJournal of AArthroscopArthroscopySports Med

Received DAddress c

OrthopedicsOutpatientjoshuaharris

� 2017 b0749-8063http://dx.d

1594

Research Pearls: The Significance ofStatistics and Perils of Pooling.

Part 3: Pearls and Pitfalls of Meta-analysesand Systematic Reviews

Joshua D. Harris, M.D., Jefferson C. Brand, M.D., Mark P. Cote, P.T., D.P.T., M.S.C.T.R., andAman Dhawan, M.D.

Abstract: Within the health care environment, there has been a recent and appropriate trend towards emphasizing thevalue of care provision. Reduced cost and higher quality improve the value of care. Quality is a challenging, heteroge-neous, variably defined concept. At the core of quality is the patient’s outcome, quantified by a vast assortment ofsubjective and objective outcome measures. There has been a recent evolution towards evidence-based medicine in healthcare, clearly elucidating the role of high-quality evidence across groups of patients and studies. Synthetic studies, such assystematic reviews and meta-analyses, are at the top of the evidence-based medicine hierarchy. Thus, these investigationsmay be the best potential source of guiding diagnostic, therapeutic, prognostic, and economic medical decision making.Systematic reviews critically appraise and synthesize the best available evidence to provide a conclusion statement (a“take-home point”) in response to a specific answerable clinical question. A meta-analysis uses statistical methods toquantitatively combine data from single studies. Meta-analyses should be performed with high methodological qualityhomogenous studies (Level I or II) or evidence randomized studies, to minimize confounding variable bias. When it isknown that the literature is inadequate or a recent systematic review has already been performed with a demonstration ofinsufficient data, then a new systematic review does not add anything meaningful to the literature. PROSPERO regis-tration and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines assist authors inthe design and conduct of systematic reviews and should always be used. Complete transparency of the conduct of thereview permits reproducibility and improves fidelity of the conclusions. Pooling of data from overly dissimilarinvestigations should be avoided. This particularly applies to Level IV evidence, that is, noncomparative investigations.With proper technique, systematic reviews and meta-analyses have the potential to be powerful investigations thatefficiently assist clinicians in decision making.

uston Methodist Orthopedics and Sports Medicine (J.D.H.),as; Heartland Orthopedic Specialists (J.C.B.), Alexandria, Min-n Musculoskeletal Institute, Human Soft Tissue Research Lab-nn Health (M.P.C.), Farmington, Connecticut; and Penn Statee and Joint Institute (A.D.), Hershey, Pennsylvania, U.S.A.rs report the following potential conflicts of interest or sources of.H. received support from Arthroscopy: The Journal ofic and Related Surgery, Smith & Nephew, Magellan, DepuySLACK Inc. J.C.B. received support from Arthroscopy: Therthroscopic and Related Surgery. A.D. received support from

y: The Journal of Arthroscopic and Related Surgery, theAssociation of North America, the Orthopaedic Journal oficine, Biomet, and Smith & Nephew.ecember 6, 2016; accepted January 31, 2017.orrespondence to Joshua D. Harris, M.D., Houston Methodistand Sports Medicine, Orthopedic Surgery, 6445 Main Street,Center, Suite 2500, Houston, TX 77030, U.S.A. E-mail:md@gmail.comy the Arthroscopy Association of North America/161227/$36.00oi.org/10.1016/j.arthro.2017.01.055

Arthroscopy: The Journal of Arthroscopic and Related Su

he delivery of health care is a highly intricate

Tprocess involving patients, providers, payers, andpolicy makers. Within the health care environment,there has been a recent and appropriate trend towardsemphasizing the value of care provision. Simplydefined, value is the quality of care per unit of cost.1

While cost can be easily defined and quantified, thequality of care is a challenging, heterogeneous, variablydefined concept. At the core of quality is the patient’soutcome, which may be quantified by a vast assortmentof subjective and objective outcome measures. Thesemeasures generate scores that can be used to compare apatient’s pre- and postintervention health status.Properly developed, valid, reliable, and responsiveoutcome measurement tools are necessary to quantifythis status. These tools are vital components withinevidence-based medicine. Within traditional science,

rgery, Vol 33, No 8 (August), 2017: pp 1594-1602

SYSTEMATIC REVIEW TIPS AND TRICKS 1595

the focus has been inherently at the individual patientlevel and at the individual study level. However, therecent evolution towards evidence-based medicine hasclearly elucidated the role of high-quality evidenceacross groups of patients and synthesis of studies.Synthetic studies, such as systematic reviews and meta-analyses, are at the top of the evidence-based medicinehierarchy (Fig 1). Thus, if performed rigorously, theseinvestigations may be the best potential source forguiding diagnostic, therapeutic, prognostic, and eco-nomic medical decision making.

Evidence-Based Medicine Hierarchy

The Problem With the Randomized TrialdIs It Reallythe Gold Standard?In the evaluation and management of patients,

evidence-based medicine should be used to guidetreatment options, discussions, and decisions. A welldesigned, conducted, and reported randomizedcontrolled trial has long been revered as the gold stan-dard of evidence for evaluating the effect of an inter-vention. This type of study is designed to produce validresults by limiting bias and confounding through the useof techniques including randomization and blinding.However, no study is without flaws or limitations.Recent investigations have demonstrated that a singlerandomized controlled trial may not produce reliableresults. There are several reasons why a single ran-domized clinical trial may reach a different conclusionthan other similar studies. Flexible statistical approaches,selective reporting, industry funding, trials that arestopped early on account of observing large positive ef-fects (overestimation of the effect), financial and nonfi-nancial conflicts of interest, and differences in patientpopulations from one to study to another are but a fewreasons that drive inconsistency and inflate treatment

Fig 1. Evidence-based medicine study hierarchy. Syntheticreviews include systematic reviews and meta-analyses and areat the top of the pyramid.

effects.2 The challenge of irreproducibility is what makessynthesis of multiple publications very useful.Replication of study results is a fundamental activity

in quantitative research.2 Studies reporting positivefindings often contradict one another (e.g., [1] eggs aregood for you, then they’re not; [2] red wine isunhealthy, then it’s not; [3] vitamin C cures the com-mon cold, then it doesn’t).3 In orthopaedic surgery,clavicle fractures should be treated nonsurgically,4 thenthey should be treated surgically5; humerus fracturesshould be treated nonsurgically,6 then they should betreated surgically.7 These results highlight the need forsimilarly designed and executed studies to confirm orrefute novel findings.The above reasons provide support for well-executed

systematic reviews and meta-analyses. The power ofthe systematic review lies in its ability to statisticallycombine patient outcomes from distinct, yet similar,research studies. This process allows the consistency, orlack thereof, of randomized trials to be examined andquantified. When executed correctly, the “true” effectof an intervention can be estimated with more precisionthan with a single trial. Equally importantly, reasons forinconsistencies (selective reporting, poor study design)and differences in the treatment effect among sub-groups can be explored. Thus, the potential exists toprovide more information than single high-qualitystudies and therefore make more powerful evidence-based conclusions. However, the quality and strengthof the recommendations from a systematic review areonly as strong as the quality of the individual studiesincluded in the analysis. Just as with randomizedcontrolled trials, great care must be exercised by journalreviewers and editors in the peer review process andreaders of the publication in interpretation of the biasand extrapolation of the review’s findings to translationto clinical practice. Given the recent rapid proliferationof both written and electronic publication outlets, well-done systematic reviews and meta-analyses are highlyuseful in their analysis and presentation of large bodiesof evidence to busy clinicians unable to peruse theentire body of literature because they can answer aclinical question in a short amount of time (Fig 2).8 Infact, systematic reviews are used by the AmericanAcademy of Orthopaedic Surgeons as the evidence tosupport creation of clinical practice guidelines andappropriate use criteria for common clinical conditionsin orthopaedic surgery.9

Benefits of Systematic ReviewsSystematic reviews critically appraise and synthesize

the best available evidence to provide a conclusionstatement (a “take-home point”) in response to a specificanswerable clinical question. The execution of asystematic review must be transparent, so that any per-son (not just an author, scientist, researcher, clinician,

Fig 2. The number of system-atic reviews (A) over the past20 years has significantlyincreased, as has the numberof meta-analyses (B).

1596 J. D. HARRIS ET AL.

etc.) can replicate the steps and arrive at the sameconclusion. Explicit eligibility criteria, with similar studyselection, inclusion, exclusion, evaluation, and reportingcriteria, allow any author to identify the same studiesfrom a search strategy as any other author. PRISMA(Preferred Reporting Items for Systematic Reviews andMeta-Analyses) guidelines were originally published in2009 in order to provide a minimum set (n ¼ 27) ofevidence-based medicine items focusing on the design,conduct, and reporting of systematic reviews.10 One ofthe items in the PRISMA checklist ensures a full trans-parent search strategy, in at least one database, by morethan one author, duplicating the same studies and withdisagreements resolved by consensus agreement with atleast one more team member.11

A meta-analysis, by definition, uses statisticalmethods to quantitatively combine data from singlestudies. Absolute inclusion of all relevant articles is nota mandatory requirement of a meta-analysis (as in asystematic review), only mathematical assimilation of 2or more studies. In other words, some meta-analysesare systematic reviews, but not all. Similarly, not allsystematic reviews are meta-analyses unless theyidentify, include, and analyze all relevant studiesquantitatively. Greater emphasis on individual studysimilarity will potentially reduce the number of eligiblestudies and subjects analyzed. This specificity improvesinternal validity of the review. Permission to conductmore dissimilar heterogeneous studies will likely in-crease the number of studies and subjects analyzed with

SYSTEMATIC REVIEW TIPS AND TRICKS 1597

improved external validity and generalizability. Thelatter concepts are analogous to efficacy (homogenousparticipants, interventions, comparators, and outcomemeasures [PICO principle] in an ideal clinical situation[analogous to a systematic review with homogenousstudies]) and effectiveness (heterogeneous, morepractical, “real world” studies in normal clinical condi-tions likely encountered in practice) clinical trials. Athorough description of all the techniques in systematicreview performance and publication is beyond thescope of this manuscript.8 The purpose of the currentmanuscript is to illustrate the pearls and pitfalls ofsystematic reviews and meta-analyses (Table 1).The goal of the meta-analysis is to pool data across

multiple studies with similar designs. Most authors,editors, and journals agree that meta-analysis should beperformed with Level I evidence studies only. In Level Iinvestigations, subjects are randomized to receive oneof 2 therapies (treatment A vs treatment B). Random-ization permits the inclusion and comparison of subjectswith different background characteristics. In meta-analyses of randomized studies, even if the subjects inone investigation (mean age, 35 years) had differentbaseline characteristics than subjects in a separateinvestigation (mean age, 55 years), randomizationwithin the individual studies permits the comparison ofthese 2 seemingly dissimilar trials since the effect oftreatment A versus treatment B may be the same inyounger and older patients. In these studies, tests forheterogeneity are focused on whether the differencesbetween treatment A and treatment B are consistent ineach study. When randomization does not occur, thevalidity of the investigation’s conclusion is significantlycompromised. If subjects within a single randomizedtrial are dissimilar across groups (treatment group A,mean age of 35 years; vs treatment group B, mean ageof 55 years), then detection of a significant differencein the outcome of interest may be based on the

Table 1. Pearls and Pitfalls of Systematic Reviews and Meta-anal

Pearls

� Register review on PROSPERO � Having a non� Follow PRISMA guidelines, checklist � Writing a ma� Identify an answerable question � Underestimat� Improve upon existing reviews � Lacking an a� Follow PICOS strategy � Failing to reco� Use 2 or more databases � Making claim� Use 2 or more reviewers� Use data extraction checklist (CEBM, Cochrane)� Generate a recommendation, its strength (SORT),

and grade the evidence (GRADE)� Generate a take-home point conclusion statement

CEBM, Centre for Evidence Based Medicine; GRADE, Grading of ReParticipants, Interventions, Comparators, Outcomes, Study Design; PMeta-Analyses; SORT, Strength of Recommendation Taxonomy.

intervention itself or the difference in age betweengroups. In a single investigation, statistical methods canaccount and adjust for these differences. In meta-analyses, since the data used are from the publicationitself, rather than the study’s raw data, it is morechallenging to use the adjusted data, as the unadjusteddata are frequently all that is presented.

Risks of Systematic ReviewsAs the evidence base of arthroscopic surgery and

related research rapidly grows and evolves, the burdenof knowledge acquisition and use in clinical practice isoverwhelming with information overload.12 Despitethe significant increase in the quantity of literature,frequently observed conclusions of systematic reviewsare “more evidence is needed,” “insufficient evidence toaddress the study purpose,” or “future evidence shouldaddress these limitations.” Thus, despite authors’ at-tempts to answer the question posed at study inception,no answer was obtained. In fact, a recent systematicreview has even been published with 0 (n ¼ 0) studiesincluded for analysis.13 Collins, Ward, and Youmattempted to answer the question, “Is prophylacticsurgery for femoroacetabular impingement indi-cated?”13 They found no trials that met criteria for in-clusion in the review and concluded that there was alack of evidence to support surgical intervention andthat future research is needed to better clarify surgicalindications. Unfortunately, this is a common studyconclusion with systematic reviews. These inconclusivereviews do not improve clinical decision making. Notonly an insufficient quantity of evidence, but also aninadequate quality of evidence or poor review meth-odology can jeopardize the impact of the review.Systematic reviews are only as good as the studies theyanalyze. The “garbage in, garbage out” phrase hasmeaning here. A systematic review of 15 Level Ievidence randomized controlled studies and one Level

yses

Pitfalls

specific, too broad, unanswerable questionnuscript that lacks sufficient conduct and design detail (transparency)ing the amount of time necessary to start and finish the reviewpriori study purpose, without explicit inclusion and exclusion criteriagnize and report study heterogeneity, limitations, biasess that extend beyond the scope of the actual results

commendations, Assessment, Development, and Evaluation; PICOS,RISMA, Preferred Reporting Items for Systematic reviews and

1598 J. D. HARRIS ET AL.

IV evidence retrospective case series is still a Level IVevidence review. Some experts contend that meta-analyses should only analyze homogeneous Level Ievidence data, as the robust statistics necessary tocombine studies is inaccurate in lower levels of evi-dence and/or more heterogenous study populations.14

This has prompted authors to better control theireligibility inclusion criteria of the review.In addition to information overload from primary

original research investigations, some authors feel thatsystematic reviews may be overly prevalent.14 When itis known that the literature is inadequate or a recentsystematic review has already been performed withdemonstration of insufficient data, then a newsystematic review does not add anything meaningful tothe literature. Recently, 3 orthopaedic journals (theJournal of Bone and Joint Surgery, American volume;Clinical Orthopaedics and Related Research; and Journal ofPediatric Orthopaedics) have published systematic reviewexpectations to (1) identify any similar previousreviews, (2) justify why a new investigation is unique,and (3) justify why the new investigation’s findings aredifferent from previous reviews. Additionally, PRISMAguidelines recommend open registration to PROSPEROof any new systematic review prior to studycommencement, to avoid duplication of resourcesdevoted to a specific clinical topic.15

PearlsIn addition to those mentioned previously, there are

several tips to improve the quality of a systematic re-view and increase its odds of publication. First andforemost, the authors must ensure that their purpose isactually achievable with a discrete answer. The authorsshould pose an “answerable question” that has ananswer of “yes”, “no”, or “unable to be determined”(pearl no. 1). The answer may also be a discrete nu-merical or categorical value(s). Thus, when the reviewis complete, and the reader has read the manuscript, atake-home point conclusion can be reported, read,understood, and remembered. Examples of answerablequestions include “Does vitamin C cure the commoncold?” “Does running cause arthritis of the knee?”“Does smoking cause lung cancer?” These all have ananswer of “yes”, “no”, or “unable to be determined”.Someone could read the conclusion in the abstract andknow the entire study in less than 10 seconds. Exam-ples of poor questions to pose for a systematic reviewinclude “What are the outcomes of total knee replace-ment?” “What is the best suture anchor?” “What arethe best treatments for osteoporosis?” Vague, nonde-script, nonspecific questions make it challenging toproduce a good take-home point for the reader to graspthe study’s conclusions (pitfall no. 1).Once an answerable question has been posed, the

authors must check to see whether the review has been

done before to ensure they are not “reinventing thewheel” and duplicating others’ recent work (pearl no.2). PROSPERO registration can help prevent this (pearlno. 3). Next, the authors should create a PRISMAchecklist for their study and start checking items off the27-item list as they progress (pearl no. 4). This willensure the integrity of the design and conduct of thereview. Following PICOS (Participants, Intervention(s),Comparator(s), Outcome(s), and Study Design) canassist the authors with the salient ingredients to besought during the search (pearl no. 5). Meticulous se-lection of PICOS criteria is what distinguishes a sys-tematic review from a simple narrative review. Clearinclusion and exclusion criteria will help ensure thatthe final studies identified for inclusion and analysis arethe ones appropriate to answer the study purpose. Aminimum of 2 databases should always be used withany review16-18 (pearl no. 6). There are several dozennonmutually exclusive electronic publicly available freedatabases to be used by researchers across the world inEnglish and non-English languages. Pay-per-use data-bases are also available and are variable in price perindividual and/or institution. The combination ofMEDLINE, Embase, and the Cochrane Central Registerof Controlled Trials generates a recall rate of at least97% in identification of all relevant studies in ortho-paedic surgery meta-analyses.19 Using 2 or morereviewers to perform the search citation strategy canminimize the risk of eligible study omission (pearl no.7). Use of a data recording checklist can be done on acustom, study-by-study individual list or file. However,the Cochrane Collaboration and CEBM (Centre forEvidence-Based Medicine) have their own generic liststhat can be used freely to assist with data collection andanalysis (pearl no. 8).After the data have been collected, the decision as to

whether it is appropriate to combine the data of theincluded studies needs to be made. This decision shouldbe made prior to conducting the review and be largelybased on how similar the studies are. Combining theresults of studies that are obviously clinically diverse orat a high and differential risk of bias may result inmisleading findings. If studies are too diverse to becombined, efforts should be made to describe whatfactors, be it clinical (very different study populations)or methodological (selective reporting, lack of blindingin some studies, etc.), are contributing to heteroge-neous group of studies. This approach helps the reviewauthor provide meaningful information regarding thecurrent research on the topic and avoid the less thaninformative conclusion of “more evidence needed.”If the studies are deemed appropriate to combine and

statistical analysis is performed, the researcher mustanswer the answerable question, confirm or reject theirstudy hypothesis(es), and come up with the study’stake-home point (pearl no. 9). Often, busy clinicians

Table 2. List of Quantitative Study Methodological QualityScores

� CLEAR-NPT� Cochrane Bone, Joint, and Muscle Trauma Group’s Methodological

Quality Score� Coleman (and modified) Methodology Score� CONSORT� Delphi List� Detsky Quality Assessment Scale� Jadad score� Newcastle-Ottawa Quality Assessment Scale� Quality Appraisal Tool� STROBE Statement� QUADAS� STARD

CLEAR-NPT, CheckList to Evaluate a Report of aNon-Pharmacologic Trial; MINORS, Methodological Index forNon-Randomized Studies; QUADAS, Quality Assessment ofDiagnostic Accuracy Studies; STARD, Standards for Reporting Studiesof Diagnostic Accuracy); STROBE, Strengthening the Reporting ofObservational Studies in Epidemiology; for study reporting, not designor conduct.

SYSTEMATIC REVIEW TIPS AND TRICKS 1599

only have time to read an abstract or even just theabstract’s conclusions. This limited text is the authors’opportunity (sometimes one and only) to convey thekey findings of the manuscript. As with all study con-clusions, this is not the time for speculation or discus-sion. The conclusions should be based wholly on the

Fig 3. Anatomy of a forest plot. CI, confidence interval; OR, odd

data, the results, the numbers. In addition, each study’smethodological quality can be numerically quantifiedby a number of different questionnaires, specific to thestudy type (therapeutic, diagnostic, prognostic,economic, randomized, observational; Table 2), tograde the strength of the study’s take-home point.These quality evaluation tools describe the potentialsources of bias (selection, performance, detection,transfer, publication, study design) in each study. Basedon the quantity, quality, and consistency of evidence, arecommendation may be made to the reader usingGRADE (Grading of Recommendations Assessment,Development, and Evaluation)20 and SORT (Strengthof Recommendation Taxonomy21; pearl no. 10). Het-erogeneity, that is, inconsistency in the treatment effector estimate across primary studies, should be assessed tocompare each study’s results with the pooled estimate.Typically heterogeneity among trials is tested with theCochrane’s Q (chi-square test) and results in an I2 sta-tistic that scores between 0 and 100% with less than25% considered low, 25% to 50% moderate, andgreater than 75% high heterogeneity.22

For a meta-analysis, the data for each variable arepresented in a forest plot listing each investigationfrequently in chronological order from the earliest atthe top to most recent at the bottom (Figs 3 and 4).23

The weighted mean is displayed as an odds ratio with

s ratio. Reproduced with permission from Cote et al.23

Fig 4. In both forest plots in panels A and B, the diamond (summary estimate) touches the line, indicating no statisticallysignificant difference. In panel A, the boxes (studies) are consistent in their findings because they each hover close to the line. Inpanel B, the boxes are much more variable. The 2 most heavily weighted studies have opposite results. The remaining studies arealso opposite in their findings. Whereas both panels A and B have the same summary estimates, panel B indicates a much morevariable group of studies. Reproduced with permission from Cote et al.23

1600 J. D. HARRIS ET AL.

a 95% confidence interval depicting whether the meanfavors the treatment or control group. The bottom rowis the combined data for the variable, for all in-vestigations. The area of each square is proportional tothe investigation’s weight in the meta-analysis. Thevertical line in the center demarcates no effect oftreatment. The right-hand column includes the oddsratio with the 95% confidence intervals. Thisdiagrammatic illustration allows the reader to quicklyassess the data for each investigation and for allinvestigations combined.Funnel plots can be included as part of a meta-

analysis and are a means to evaluate publication bias,that is, the likelihood of small investigations with apositive result to be published and investigationswithout a difference between the treatment and controlgroup to not be published (Fig 5).23 Sample size or

Fig 5. (A) A funnel plot without any evidence of publication biahover close to the summary estimate, whereas smaller studies are spotential publication bias. The missing area of the funnel may be sthence underpowered. Either by way of rejection from a journastudies in theory may exist and, if published, would have been inReproduced with permission from Cote et al.23

standard error, on the y axis, is plotted against estimateseither favoring treatment or control of the in-vestigations, on the x axis. Funnel plots are scatterplotsthat demonstrate bias or systematic heterogeneity. Asymmetric funnel plot ideally looks like an upside-down funnel with symmetric data distribution,making publication bias unlikely.

PitfallsCommon mistakes during design, performance, and

reporting of systematic reviews can largely be avoidedby following the pearls and tips mentioned earlier.However, in addition, authors should strive to ensurecomplete transparency and details in the initial manu-script submission prior to publication. This pitfall ofomission can be avoided during the data extractionstage by simply “recording everything”8 (pitfall no. 2). It

s. The largest studies are located at the top of the graph andcattered about the bottom of the graph. (B) A funnel plot withudies that were small in size and found no difference and werel or by simply never making into a submission queue, thesecluded and perhaps swayed the findings of the meta-analysis.

SYSTEMATIC REVIEW TIPS AND TRICKS 1601

is much easier to record everything initially, reportexactly what you did in the review’s methods section,and then reduce if needed based on journal re-quirements or word counts. This will ensure that anyperson (not just an author, reviewer, editor, scientist,author) can replicate the review’s methods and arriveat the exact same conclusion. This last statementimplies that the process may take a significant amountof time. On account of the increasing prevalence ofsystematic reviews, many authors, journals, and editorshave discounted, to variable degrees, the amount oftime and effort required to produce a high-qualitysystematic review. Nearly all systematic review au-thors underestimate the amount of time necessary forreview completion (pitfall no. 3). Despite the fact thatsystematic reviews do not require Institutional ReviewBoard approval, funding, patient recruitment, contact,or follow-up, the time commitment is significant.A significant pitfall commonly observed is a brief

limitations section (pitfall no. 4). Every study, system-atic reviews included, has biases. This especially isrelevant for quantitatively evaluating the methodolog-ical quality of the review. These (Table 2) question-naires assist authors in how to better properly designfuture research to help improve the quality. In fact,quantitative grading of the quality of systematic reviewscan be done using AMSTAR (Assessment of MultipleSystematic Reviews)24 and MECIR (MethodologicalExpectations of Cochrane Intervention Reviews),25

which, in turn, help systematic review authors indesign, conduct, and reporting of future reviews.A frequent pitfall seen in the manuscripts received at

Arthroscopy is the pooling of investigations with differentmethods (pitfall no. 5). If different patient-reportedoutcome scores are measured for what is already asmall number of included investigations (for Arthros-copy, we recommend that more than 10 investigationsmake up a systematic review or meta-analysis), thecomparisons may be limited to 2 or 3 investigations fora single variable. This adds little value to clinical deci-sion making. If different surgical techniques areincluded in the meta-analysis, yet the authors concludethat surgery improves measured outcomes, then again,little additional clarity regarding specific treatment forthe individual patient is gained by the clinician.Systematic reviews or meta-analyses that comparetreatment to a control should include both treatmentarms. Thus, at a minimum, Level III investigations orhigher (I or II) should be included for comparison trials.Including Level IV investigations without comparabletreatment arms may introduce bias and heterogeneity.Many Level IV studies do not measure a specific treat-ment effect. Thus, comparison of 2 or more studies likethis involves significant differences in interventiontechnique, rehabilitation, subject demographics, andoutcome measures, among several others. These

differences invalidate comparison of these studies.Nevertheless, a frequently observed error, especially inLevel IV evidence reviews, is the inappropriately“compared” heterogeneous studies that reach conclu-sions that extend beyond those of the actual resultsreported (pitfall no. 6). The authors’ conclusions shouldalways be based on observed results, not discussion,extrapolation, interpretation, possibilities, potential, orfuture research.

ConclusionsMeta-analyses (at the top) and systematic reviews

(second from the top) are powerful studies at the top ofthe evidence-based medicine hierarchy. They have theability to precisely select eligible studies, participants,relevant interventions, outcomes of interest, andpotential comparator groups. Outcomes can be quan-titatively assimilated and compared (meta-analysis),increasing subject numbers and statistical power.Proper inclusion and exclusion criteria can mitigate, butnot eliminate, review bias. As with all investigations, aproper conclusion should be a take-home point thatanswers an answerable question based on the actualstudy results and data.

References1. Porter ME. What is value in health care? N Engl J Med

2010;363:2477-2481.2. Ioannidis JP. Why most discovered true associations are

inflated. Epidemiology 2008;19:640-648.3. Schoenfeld JD, Ioannidis JP. Is everything we eat

associated with cancer? A systematic cookbook review.Am J Clin Nutr 2013;97:127-134.

4. Neer CS 2nd. Nonunion of the clavicle. JAMA 1960;172:1006-1011.

5. Altamimi SA, McKee MD, for the Canadian OrthopaedicTrauma Society. Nonoperative treatment compared withplate fixation of displaced midshaft clavicular fractures. Amulticenter, randomized clinical trial. J Bone Joint Surg Am2007;89:1-10.

6. Sarmiento A, Kinman PB, Galvin EG, Schmitt RH,Phillips JG. Functional bracing of fractures of the shaft ofthe humerus. J Bone Joint Surg Am 1977;59:596-601.

7. Canavese F, Marengo L, Cravino M, et al. Outcome ofconservative versus surgical treatment of humeral shaftfracture in children and adolescents: comparison betweennonoperative treatment (Desault’s bandage), externalfixation and elastic stable intramedullary nailing. J PediatrOrthop 2017;37:e156-e163.

8. Harris JD, Quatman CE, Manring MM, Siston RA,Flanigan DC. How to write a systematic review. Am JSports Med 2014;42:2761-2768.

9. AAOS. Clinical Practice Guidelines. 2016. http://www.aaos.org/Quality/Clinical_Practice_Guidelines/Clinical_Practice_Guidelines/. Accessed November 1, 2016.

10. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMAstatement for reporting systematic reviews andmeta-analyses of studies that evaluate health care

1602 J. D. HARRIS ET AL.

interventions: explanation and elaboration. J Clin Epi-demiol 2009;62:e1-e34.

11. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferredreporting items for systematic reviews and meta-analyses:the PRISMA statement. J Clin Epidemiol 2009;62:1006-1012.

12. Lubowitz JH, Brand JC, Provencher MT, Rossi MJ.Systematic reviews keep arthroscopy up to date. Arthros-copy 2016;32:237.

13. Collins JA, Ward JP, Youm T. Is prophylactic surgery forfemoroacetabular impingement indicated? A systematicreview. Am J Sports Med 2014;42:3009-3015.

14. Provencher MT, Brand JC, Rossi MJ, Lubowitz JH. Areorthopaedic systematic reviews overly prevalent?Arthroscopy 2016;32:955-956.

15. PROSPEROdInternational Prospective Register of System-aticReviews.2016.http://www.crd.york.ac.uk/PROSPERO/.Accessed November 1, 2016.

16. HopewellS,ClarkeM,LefebvreC,SchererR.Handsearchingversus electronic searching to identify reports of randomizedtrials. Cochrane Database Syst Rev 2007:Mr000001.

17. Suarez-Almazor ME, Belseck E, Homik J, Dorgan M,Ramos-Remus C. Identifying clinical trials in the medicalliterature with electronic databases: MEDLINE alone isnot enough. Control Clin Trials 2000;21:476-487.

18. Whiting P, Westwood M, Burke M, Sterne J, Glanville J.Systematic reviews of test accuracy should search a rangeof databases to identify primary studies. J Clin Epidemiol2008;61:357-364.

19. Slobogean GP, Verma A, Giustini D, Slobogean BL,Mulpuri K. MEDLINE, EMBASE, and Cochrane indexmost primary studies but not abstracts included in or-thopedic meta-analyses. J Clin Epidemiol 2009;62:1261-1267.

20. Grading of recommendations assessment, development,and evaluation. 2000. http://gradeworkinggroup.org/.Accessed November 1, 2016.

21. Ebell MH, Siwek J, Weiss BD, et al. Strength of recom-mendation taxonomy (SORT): a patient-centeredapproach to grading evidence in the medical literature.J Am Board Fam Pract 2004;17:59-67.

22. Zlowodzki M, Poolman RW, Kerkhoffs GM,Tornetta P 3rd, Bhandari M. How to interpret ameta-analysis and judge its value as a guide for clinicalpractice. Acta Orthopaedica 2007;78:598-609.

23. Cote MP, Apostolakos JM, Voss A, DiVenere J,Arciero RA, Mazzocca AD. A systematic review of meta-analyses. Arthroscopy 2016;32:528-537.

24. Shea BJ, Hamel C, Wells GA, et al. AMSTAR is a reliableand valid measurement tool to assess the methodologicalquality of systematic reviews. J Clin Epidemiol 2009;62:1013-1020.

25. Methodological standards for the conduct of newCochraneIntervention Reviews. Version 2.2. 2012. http://www.editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/files/uploads/MECIR_conduct_standards%202.2%2017122012.pdf. Accessed December 22, 2012.