Systematic Reviews and the American Academy of Pediatrics

Virginia A. Moyer, MD, MPH

Professor of Pediatrics

Baylor College of Medicine

American Academy of Pediatrics

Founded in 1930 as an independent pediatric forum to address children’s needs

60,000 members worldwide 34,000 certified by the American Board of Pediatrics and

designated as Fellows of the AAP The mission of the American Academy of Pediatrics is to

attain optimal physical, mental, and social health and well-being for all infants, children, adolescents, and young adults. To accomplish this mission, the Academy shall support the professional needs of its members.

Categories of AAP Policy

Policy Statement - A statement of advocacy, direction, or a public health position of concern to the Academy, including recommendations

Clinical Report - The Clinical Report (CR) offers guidance for the pediatrician in the clinical setting, addressing best practices and state of the art medicine, not supposed to include formal recommendations.

Clinical Practice Guideline - Based on a comprehensive literature review and data analyses with formal rules of evidence in support of each recommendation made.

Technical Report - Based on a literature review and data analyses but does not contain recommendations.

Methodology for CPG Development

“As far as possible, conclusions about optimal management and recommendations to the clinician will be based on a rigorous analysis of current literature”

Rules of Evidence

“Authoring entities should consider using formal rules of evidence in the development of the document, following the format of the US Preventive Services Task Force Guide to Clinical Preventive Services, 2nd Edition (1996).”

Aggregate Evidence Quality

Next, guideline developers must consider the quality of the aggregate of studies that bear on the issue. Judging the strength of a body of evidence requires careful consideration of the consistency of the results of individual studies, the magnitude of the effect that the studies detect, and the individual and aggregate sample sizes of these studies.

West S, King V, Carey TS, et al. Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment no. 47. Research Triangle Park, NC: Research Triangle Institute-University of North Carolina Evidence-Based Practice Center; 2002. AHRQ Publication no. 02-E016

Grading Recommendation Strength

Evidence Quality

Preponderance of Benefit or

Harm

Balance of Benefit and

Harm

A. Well designed RCTs or diagnostic studies on relevant population

B. RCTs or diagnostic studies with minor limitations;overwhelmingly consistent evidence from observational studies

C. Observational studies (case-control and cohort design)

D. Expert opinion, case reports, reasoning from first principles

X. Exceptional situations where validating studies cannot be performed and there is a clear preponderance of benefit or harm

Strong

Strong

Rec

RecOption

Option No Rec

Classifying Recommendations for Clinical Practice Guidelines. Pediatrics 2004;114(3):874-877

The Reality of AAP SRs

“A thorough review of all relevant databases with standardized search terms is required;  We do not always require more than one reviewer in rating the article- we should, but it's not always feasible.  RCTs are better than observational studies are better than case reports/first principles…  Articles outside of the years searched are not allowed to be included in determining the strength of evidence/recommendation, but can be cited in supporting text.  If an article comes out that will change the strength, then we try to update the full lit review.”

In-house SRs are internally funded, but available funding is limited, which may significantly affect quality and timeline.

Personal communication – CPG Manager

EPC evidence reviews

“The TEP was well rounded and included a number of members who are on the AOM guideline committee… I have also been in frequent communication with AHRQ to help move the report along while we simultaneously began the guideline process using the AHRQ draft… Unfortunately there were numerous delays. Administratively the draft sat at AHRQ for several months before it was put out for comment. Several members of the guideline committee including those on the TEP had problems with the draft. Currently I believe the draft is undergoing significant changes and we do not know when we will see newer versions and a final product… Since we need the AHRQ report for our evidence base, we will not meet our target date. In addition we will have to attempt to review new literature to keep the guideline up to date.”

“As of right now, it's at least 2 years from topic submission till there's an evidence review, and then another 2 years till there's a guideline.  It doesn't make the committees very receptive to writing guidelines.”

Chairs, Guideline Committees

AAP “Endorsed” Guidelines

Use COGS checklist to assess quality COGS has some SR assessment

questions

Looking ahead

Increasing need for high quality evidence reviews Comprehensive reviews for CPGs Limited scope reviews for Policy Statements and Clinical

Reports Significant cost and limitations of in-house evidence

reports for CPGs Significant time issues with external evidence reports for

CPGs Significant methodology issues for limited scope reviews AAP would welcome uniform guidelines, but these need

to be practical and applicable to multiple situations.