Improving the Quality of Immunization Delivery to an At-Risk Population: A Comprehensive Approach

abstractOBJECTIVE: Immunization quality improvement (QI) interventions arerarely tested as multicomponent interventions within the context ofa theoretical framework proven to improve outcomes. Our goal wasto study a comprehensive QI program to increase immunization ratesfor underserved children that relied on recommendations from theCenters for Disease Control and Prevention’s Task Force on Commu-nity Preventive Services and the framework of the Chronic CareModel.

METHODS: QI activities occurred from September 2007 to May 2008 at 6health centers serving a low-income, minority population in Washington,DC. Interventions included family reminders, education, expandingimmunization access, reminders and feedback for providers, andcoordination of activities with community stakeholders. We determinedproject effectiveness in improving the 4:3:1:3:3:1:3 vaccination series(4 diphtheria-tetanus-pertussis vaccines, 3 poliovirus vaccines, 1measles-mumps-rubella vaccine, 3 Haemophilus influenzae typeb vaccines, 3 hepatitis B vaccines, 1 varicella vaccine, and three7-valent pneumococcal conjugate vaccines) compliance.

RESULTS: We found a 16% increase in immunization rates overall anda 14% increase in on-time immunization by 24 months of age.Improvement was achieved at all 6 health centers and maintainedbeyond 18 months.

CONCLUSION: We were able to implement a comprehensive immu-nization QI program that was sustainable over time. Pediatrics2012;129:e496–e503

AUTHORS: Linda Y. Fu, MD, MS,a Mark Weissman, MD,a,b

Rosie McLaren, MS,c,d Cherie Thomas, BSN, RN,d JacquelynCampbell, MSN,d Jacob Mbafor, MS,d Urvi Doshi, BS,d andDenice Cora-Bramble, MD, MBAa

aGoldberg Center for Community Pediatric Health, Children’sNational Medical Center, Washington, DC; bDC Partnership toImprove Children’s Healthcare Quality, Washington, DC; cNationalCenter for Immunization and Respiratory Diseases, Centers forDisease Control and Prevention, Atlanta, Georgia; and dDistrict ofColumbia Department of Health, Washington, DC

KEY WORDSimmunizations, quality improvement, vaccines, pediatric,pediatric outpatient clinics

ABBREVIATIONSACIP—Advisory Committee on Immunization PracticesCCM—Chronic Care ModelCDC—Centers for Disease Control and PreventionCoCASA—Comprehensive Clinic Assessment Software ApplicationIIS—immunization information systemMOGE—moved or gone elsewherePCV7—7-valent pneumococcal conjugate vaccineQI—quality improvementWIC—Women, Infants and Children

www.pediatrics.org/cgi/doi/10.1542/peds.2010-3610

doi:10.1542/peds.2010-3610

Accepted for publication Oct 10, 2011

Address correspondence to Linda Y. Fu, MD, MS, Goldberg Centerfor General and Community Pediatrics, Children’s NationalMedical Center, 111 Michigan Ave, NW, Washington, DC 20010.E-mail: lfu@cnmc.org

PEDIATRICS (ISSN Numbers: Print, 0031-4005; Online, 1098-4275).

Copyright © 2012 by the American Academy of Pediatrics

FINANCIAL DISCLOSURE: Drs Fu and Cora-Bramble have servedon an advisory board to Pfizer on reducing barriers toimmunization among special populations. Dr Cora-Bramble hasalso been a reviewer of Pfizer grants with this same purpose. Inaddition, for 3 months, our health centers participated ina program sponsored by Sanofi-Pasteur to receive a reducedrate for placing autodialer calls.

e496 FU et al by guest on August 13, 2015pediatrics.aappublications.orgDownloaded from

Nearly 1 in 4 young children in theUnited States are not appropriatelyimmunized.1 Underimmunized childrenare more likely to be poor,2,3 fromsingle-parent households,2 AfricanAmerican,3–6 and from inner cities.7–9

The Centers for Disease Control andPrevention’s (CDC’s) Task Force onCommunity Preventive Services hasrecommended 13 interventions withthe potential to improve immunizationrates based on a literature review oftheir efficacy.10,11 In 2008, the recom-mended interventions related to out-patient pediatric practice includedclient reminder/recall systems, re-duction of out-of-pocket costs, expan-sion of access in clinical settings,provider reminder/recall, multicompo-nent educational strategies, assess-ment and feedback for providers,standing orders, and vaccination pro-grams in Women, Infants and Children(WIC) settings. To our knowledge, nostudy has evaluated the impact ofimplementing all the Task Force rec-ommendations related to ambulatorypediatric practice jointly.

Because the Task Force recommen-dations only include proven effectivestrategies, they potentially miss bene-ficial but untested interventions. Toexamine this possibility, it is useful toconsider the fit of the Task Force rec-ommendations to a theoretical frame-work designed to improve patientoutcomes such as the Chronic CareModel (CCM). The CCM includes 6 ele-ments: community resources and poli-cies, patient self-management support,health care organization, delivery sys-temdesign, decisionsupport, andclinicalinformation systems.12,13 Implementationof the model has been shown to improvepatient outcomes for various pop-ulations, including children, and varioushealth conditions.14,15 The CCM is appli-cable to improving immunization cover-age because, similar to chronic care,immunization occurs in the outpatient

setting and requires a sustained com-mitment to preventive services by thecommunity and health care system.

Our 6 health centers instituted an im-munization quality improvement (QI)programinwhichwe implementedall ofthe Task Force’s recommended inter-ventions relevant to outpatient pediatricpractice. We fit the recommendations tothe CCM framework and decided to addanother intervention we felt was miss-ing, yet important for success. The aimof this study was to determine the ef-fectiveness of our immunization QI pro-gram for increasing compliancewith the4:3:1:3:3:1:3 vaccination series (whichincludes 4 diphtheria-tetanus-pertussisvaccines, 3 poliovirus vaccines, 1measles-mumps-rubella vaccine, 3 Haemophilusinfluenzae type b vaccines, 3 hepatitis Bvaccines, 1 varicella vaccine, and 3 dosesof 7-valent pneumococcal conjugatevaccine [PCV7]) among children 19 to35 months old.

METHODS

Setting

Activities occurred from September2007 to May 2008 at 6 health centersassociated with a pediatric hospital inWashington, DC. The largest center,center C, is colocated with the hospital,whereas the others are community-based. Center F includes a permanentsite and a mobile van. All centers servepatients who are predominantly poorand publicly insured (Table 1). The ma-jority of patients are African Americanat all centers, with the exception ofcenter B, which is predominantly His-panic. Collectively, ∼30 physicians, 3nurse practitioners, 90 residents, and42 nurses immunize ∼37% of DC’s pe-diatric population annually.

Our QI program paid staff consisted of 1physician 30% time, and 2 full-timeadministrators. The study protocol wasapproved by the Children’s NationalMedical Center institutional reviewboard.

Planning the QI Initiative

We first determined the fit of the TaskForce recommendations with the CCM(Table 2).16 Although the fit was good,there remains a deficiency of studiesexamining the interaction betweenhealth care institutions and the com-munity. According to the CCM, healthcare organization (defined as its goalsand values and relationships withother community stakeholders) is a vi-tal pillar of the chronic care edifice.Therefore, we chose to add communitypartnership to our activities.

We chose not to implement 1 Task Forcerecommendation—reducing out-of-pocket costs. This intervention did notfit well into the CCM framework andwas not an important consideration forour patients because the majority wasMedicaid enrollees who did not incurpersonal costs for immunization.

Implementing the QI Initiative

Table 2 describes the changes we im-plemented and evaluated. All changeswere first implemented at health centerC where our program director was lo-cated. We implemented each change ona small scale for 1 to 3 weeks beforeseeking feedback from staff and/orfamilies. With feedback, we altered theinterventions as needed. Project leadersmet monthly with immunization cham-pions (including physicians, nurses, andadministrators) from each health cen-ter, representatives of the DC De-partment of Health (DOH) ImmunizationProgram, and the four DC Medicaidhealth plans to share ideas and discussintervention implementation strategies.Afterward, immunization champions in-troduced the changes at their respectivecenters, tailoring them to enhance per-formance improvement for their setting.

Collaboration With CommunityStakeholders

The DC DOH helped us to identify in-correct monthly health center vaccine

QUALITY REPORT

PEDIATRICS Volume 129, Number 2, February 2012 e497 by guest on August 13, 2015pediatrics.aappublications.orgDownloaded from

orders (eg, insufficient ordering quan-tities based on last year’s monthlyorder) and common practitioner ad-ministration errors (eg, administeringthe third hepatitis B dose too soon).Collaboration with the Medicaid plansenhanced outreach. Our QI staff made 2phone calls and mailed a postcard tofamilies with children overdue forimmunizations. If these attempts wereunsuccessful, the health plans couldsometimes supply new contact orhealth practitioner information and, ifnot, could perform home visits.

Provider Reminder/Recall andAssessment and Feedback forProviders

The DC DOH Immunization InformationSystem (IIS) was used to generatereminders forour practitioners for eachpatient encounter. Individual patient IISrecords listed past immunizations andthose that were currently due andoverdue. Since 2008, reporting to the IIShas been mandatory for all DC healthcare providers, licensed daycare pro-viders,andpublic schools,makingaverycomplete registry. According to the CDCIIS Annual Report, in 2008, the IIS cap-tured 100% of DC’s childhood populationbetween 4 months and 6 years old.17

The IIS was also used to produce im-munizationcoveragesummaryreports.Monthly immunization compliance runcharts (compliance rates graphed overtime) were created for each healthcenter individually and overall. Runcharts for each centerwere sharedwith

all practitioners as process perfor-mance feedback. All practitioners par-ticipated in at least 3 training sessionsto learn how to interpret IIS records,strategize intervention implementation,and discuss run chart trends.

Expanding Access in Clinical Settings

To expand access to immunizations,practitioners at all health centers be-gan immunizing at unscheduled “sick”visits in addition to well visits. Anotherappointment was scheduled for chil-dren receiving immunizations at sickvisits who were also due for a well ex-amination. Parents were only givenhealth certificates (required for childcare and school enrollment) once thechild completed a well examination.Access to immunizations was also in-creased at health centers C and F withimmunization-only clinics (held twiceduring influenza season at center Cand monthly at center F).

Standing Orders

Because of the medical complexityof some of our patients, we choseto have a physician involved in allimmunization-only visits. As a modifiedversion of standing orders, nursescould initiate immunization orders for“shots-only visits,” but physicians ap-proved all orders before immuniza-tions were administered.

Client Reminder/Recall Systems

Wesentreminderautodialercallsmonthlyto children due for immunizations.

Becauseofour largepatientpopulation,we focused on children ,24 monthsold. To recall children overdue for im-munizations, we made least 2 livephone call attempts. When a parentanswered, we attempted to schedulean appointment. If we were unable toreach a family via the telephone, wesent a postcard. Because outreach foroverdue immunizations was labor in-tensive, we limited this activity to chil-dren ,60 months old.

Educational Interventions

We created informational posterswith pictorial examples of vaccine-preventable diseases, school immuni-zation requirements, and immunizationmyths and facts. Because physicalspace varied amongourhealth centers,centers A, B, and C placed posters in allexamination and waiting rooms,whereas D had them in the procedureroom, and E had them in the triageroom. As an additional educational in-tervention, forms were created forpractitioners to document for parentsthe immunizations their child receivedthat day and the due date for futureimmunizations.

Vaccination Programs in WIC Settings

WIC is a federally subsidized nutritionalprogram for pregnant women, infants,and children. For our intervention, weprescreened the IIS record for everyWIC client ,6 years old ahead of WICappointments at the center colocatedwith health center C. We then asked WIC

TABLE 1 Overall Health Center Sociodemographic Characteristics in 2008

Health Center

Overall A B C D E F

Annual patient visits 71 936 5507 (8%) 11 068 (15%) 30 156 (42%) 8323 (12%) 6337 (9%) 10 544 (15%)Publicly insured, n (%) 60 921 (85%) 4538 (82%) 9684 (87%) 25 104 (83%) 7302 (88%) 5532 (87%) 8761 (83%)Race/ethnicity, n (%)African AmericanHispanicWhiteAsianOther/unknown

53 264 (74%)10 885 (15%)

380 (1%)602 (1%)6805 (9%)

4721 (86%)163 (3%)34 (1%)266 (1%)323 (6%)

2132 (19%)8454 (76%)62 (1%)140 (1%)280 (3%)

22 766 (75%)2120 (7%)221 (7%)185 (1%)4860 (16%)

7992 (96%)10 (0%)6 (0%)0 (0%)

315 (4%)

5603 (88%)14 0%)27 (0%)6 (0%)

687 (11%)

10 046 (95%)123 (1%)30 (0%)5 (0%)

340 (3%)

e498 FU et al by guest on August 13, 2015pediatrics.aappublications.orgDownloaded from

TABLE2

TheDeterm

inants

ofPatient

Outcom

eas

They

Relate

toDifferent

Typesof

RecommendedImmunizationQI

Interventions

andtheChangesWeImplem

ented

TheoreticalDeterm

inantof

Patient

Outcom

e(fromtheChronicCare

Model)

Evidence-Based

Recommendation

(fromtheCDCTask

Forceon

Community

PreventiveServices)

ActualChangesImplem

ented

HowFrequentlytheInterventionWas

Used

atEach

Health

Center

AB

CD

EF

Health

care

organization

•Collaborationwith

community

stakeholders

a

•Monthlymeetings

wereheldwith

DCDepartmentofHealth

and

MedicaidMCO

representatives

todiscussanyissues

with

ordering

vaccines,docum

entingor

transferring

vaccination

datatotheIIS,coordinationofoutreach

efforts,etcb

N/A

N/A

N/A

N/A

N/A

N/A

•Names

ofchildrenoverdueforimmunizations

who

wewere

unabletocontactw

eretransferredtotheirMCOsofrecord

for

moreextensiveoutreach

(eg,homevisits).

dd

dd

d◯

Clinicalinform

ationsystem

s•Provider

reminder/recall

•IIS

vaccinationstatus

reportwas

generatedforeverypatient

encounteras

areminder/recallforproviderstoorderdueor

overduevaccines.

dd

dd

dd

Decision

support

•Assessmentand

feedback

for

vaccinationproviders

•Three

training

sessions

forallproviderswereheldduring

staff

meetings

andresidentconferencesreview

ingthevaccination

schedules,reducing

missedopportunities

forvaccinationand

improvingdocumentationofvaccines

administered.

dd

dd

dd

•Immunizationcoverage

foreach

health

center’spatient

populationwas

graphedmonthlyandshared

with

allcenters

tochartp

rogress.

dd

dd

dd

•Provider

reminder/recall

•Seeabovedescription

dd

dd

dd

Deliverysystem

design

•Expandingaccess

inclinicalsettings

•Vaccinationstatus

was

assessed

andoverdueimmunizations

administeredatsick

andwellvisits

(versusjustwellvisits).

dd

dd

dd

•Immunization-onlyclinics

◯◯

◉◯

◯d

•Standing

orders

•Nurses

verified

allphysician

vaccineorders

with

IISrecommendations,and

discrepancieswerediscussed.

dd

dd

dd

Self-managem

entsupport

•Client

reminder/recallsystem

s•Autodialer

messagesweresentmonthlytoallfam

ilies

with

2-,

4-,6-,12-,18-,and24-mo-oldchildrentoremindthem

ofdue

immunizations.

dd

dd

dd

•Staffm

adetwoattemptsatlivephonecalls

toreachfamilies

with

childrenages

0-4yoldwho

wereoverduefor

immunizations.A

postcard

was

mailedtofamilies

unreachableby

phone.Ifno

appointm

entw

asscheduled1mo

afterthepostcard

was

mailed(orpostcard

was

returned

tosender),thechild’snamewas

forw

ardedtohishealthplan

for

furtheroutreach.

dd

dd

dd

•Multicom

ponent

interventions

that

includeeducation

•Handoutswith

theimmunizationscheduleandspacefor

providerstodocumenttheduedateforthenextsetof

immunizations

weregivento

allparentsofnewbornsb

◉◉

◉◉

◉◯

•Posterswereplaced

inwaitingareasandexam

inationroom

swith

pictorialexamples

ofvaccine-preventablediseases,

schoolimmunizationrequirem

ents,aswellasimmunization

myths

andfacts.

◉◉

dd

d◯

QUALITY REPORT

PEDIATRICS Volume 129, Number 2, February 2012 e499 by guest on August 13, 2015pediatrics.aappublications.orgDownloaded from

personnel to refer underimmunizedchildren to our health center to receivesame-day immunizations immediatelyafter their WIC appointment.

Evaluation Methods

We asked immunization champions toassess the frequencywithwhichactivitieswere implemented at their site. Otherprocess measures we tracked includedthe monthly number of calls for due im-munizations thatconnected toapersonorvoicemail, and the number of phone callsmade and postcards sent for overdueimmunizations. We estimated the numberof provider reminders generated by usingbilling data to determine the number ofpatient visits that occurred in the daterange that IIS records were generated.

We measured the outcome of our QIprogram by using a before-and-afterobservational approach. Namely, theDC DOH evaluated the immunizationcoverage of our health centers (doneroutinely for all DC participants of theVaccines for Children Program) by us-ing the CDC’s Comprehensive Clinic As-sessment Software Application (CoCASA)(earlier versions known as CASA). Animmunization was considered validonly if it was administered at the rec-ommended age/interval per the AdvisoryCommittee on Immunization Practices(ACIP). Immunization rates for thehealth centers were compared at 3 timepoints: spring 2006 (preintervention),spring 2008 (after all QI activities weredeployed), and spring 2009 (18 monthspostintervention initiation). Data werefirst imported from the DC IIS intoCoCASA. For any child found to be over-due according to the IIS, the healthcenter’s medical chart was reviewed.Any additional immunizations recordedin the chart were added into CoCASA(and the IIS).

Analysis

Our primary outcome was immuniza-tion coverage for 19- to 35-month-old

children for the 4:3:1:3:3:1:3 series. Al-though ACIP recommended four dosesof PCV7 at the time, we chose the4:3:1:3:3:1:3 series because our healthcenters experienced frequent shortagesof PCV7 during the national shortage in2004.18 The 4:3:1:3:3:1:3 series was a keyseries assessed by the CDC in 2008 forchildren ages 19 to -35 months.19

Secondary outcomes were also as-sessed to determine the timeliness ofimmunization and sustainability ofresults. To examine the effect of the QIinitiative on timeliness, we comparedseries completion rates among 19- to24-month-old children in 2006 and 2008.This age range, which is 0 to 6 monthsbeyond the upper age limit of when the4:3:1:3:3:1:3 vaccines are due, is how theCDC’s Immunization Program Evalua-tion reporting requirements define “ontime.” To determine sustainability, cov-erage for 2009 was compared with im-munization coverage for 2008. Alloutcomes were considered as di-chotomous variables (series completeor not). Comparisons were made byusing the x2 test.

We performed analyses including allpatients for whom 1 of our healthcenters was the provider of record inthe IIS. We also performed analysesexcluding all patients we deemed aMOGE (moved or gone elsewhere). Wedefined MOGE as any child with docu-mentation of (1) a move out of the area,(2) a change to a health care providernot affiliated with one of the studyhealth centers, or (3) no clinical en-counter in.12 consecutive months forwhom all outreach attempts were un-successful (wrong number/phone dis-connected and postcard returned tosender/no response). The second setof analyses more accurately reflectsimmunization coverage for patientswho used our health centers for pri-mary care services versus urgent careonly, because providers in the IIS areassigned solely according to whichTA

BLE2

Continued

TheoreticalDeterm

inantof

Patient

Outcom

e(fromtheChronicCare

Model)

Evidence-Based

Recommendation

(fromtheCDCTask

Forceon

Community

PreventiveServices)

ActualChangesImplem

ented

HowFrequentlytheInterventionWas

Used

atEach

Health

Center

AB

CD

EF

Community

resourcesandpolicies

•VaccinationprogramsinWICsettings

•Vaccinationstatus

was

assessed

forWICclientsatacenter

colocatedwith

oneofourhealthcenters.WICclients#5yold

overdueforvaccines

werewalkedacross

thehallw

aytothe

health

center

toschedulean

appointm

entor

towaittosee

aphysicianthatdayc

◯◯

d◯

◯◯

d,alwaysor

very

frequently;◉

,occasionally

orrarely;◯

,veryrarelyor

never

aInterventiontype

inadditiontothe7recommendedby

theTask

Forceon

Community

PreventiveServices.

bMonthlymeetings

with

community

stakeholders

wereattended

byimmunizationQIleadershipon

behalfofallhealth

centers.

cThisinterventionwas

also

discontinuedathealth

center

Caftera3-weektrialbecause

ofminimaleffect.

e500 FU et al by guest on August 13, 2015pediatrics.aappublications.orgDownloaded from

institution administered the child’smost recent immunization. Healthcenter D, located at the main hospital,is regularly used as an urgent careclinic by children with other primarycare providers. Incorrect IIS providerassignments to health center D be-came a more common occurrencewhen we began immunizing children atsick visits (regardless of whether theyidentified other primary care pro-viders) as part of our QI program.

RESULTS

Patient Population

There were 63 524 total patient visitsoverall in 2006 vs 71 935 in 2008, anincrease of 12% (P , .0001). In 2006,49 782 (78%) patients were publiclyinsured compared with 60 921 (85%) in2008 (P , .0001).

Process Measures

In 2006, IIS recordswere not preprintedfor any patient encounters. In 2008, anIIS record was preprinted for practi-tioners’ reference for virtually everyscheduled patient encounter (41 685records) at all centers, and ∼25%(7290 records) of all unscheduled “sickvisit” encounters (records were notpreprinted for all unscheduled en-counters because of limited QI staff).

Before the study, in 2006, we did notsend any reminder/recall phone mes-sages or mailings. In 2008, we sent out27 651 autodialer messages for dueimmunizations, and made 10 253 livephone calls and sent 523 postcards foroverdue immunizations.

In 2006, personnel at WIC centers co-located with our health centers did notcheck their clients’ vaccine compliance,but did ask parents to bring immuni-zation records to every visit that theythen forwarded without review to theIIS. In 2008, during a 3-week trial, wechecked the IIS record of 370 WIC cli-ents. In this group, 21 children were

overdue for immunizations. Of these, 1child received same-day immuniza-tions, 3 parents refused, and 17 did notpresent to their WIC appointments.Because of low yield, we did not con-tinue this intervention.

Outcome Measures

In 2006, the health centers were theprimary care providers for 1999 childrenages 19 to 35 months compared with1946 children in 2008 (ie, MOGEs wereexcluded both years). Immunization cov-erage for the4:3:1:3:3:1:3 series improvedfrom 71% to 87% (P , .0001) (Table 3).Coverage increased significantly at all sixhealth centers. In addition, timely vacci-nation rates improved from 65% to 79%(P , .0001) and increased significantlyat 4 of the 6 centers, including the largestand smallest (Table 4).

We also performed data analysis withall patients for whom our centers werethe immunizer of record in the IIS (in-cludingallMOGEsasnot up todate, datanot shown). This larger data set in-cluded 2232 children aged 19 to 35months in 2006 and 2250 children in2008. Immunization coverage overallfor the 4:3:1:3:3:1:3 series improvedfrom 64% to 75% (P , .0001). Immu-nization rates increased significantly at5 of 6 health centers (P, .05) with theincrease at center E reaching border-line significance (P = .07). In addition,timely immunization rates improvedoverall from 58% to 69% (P , .0001).Rates of timely immunization increasedsignificantly at health centers B, C, D,and G (P , .05), and was borderlinesignificant at center E (P = .07).

To determine whether gains in immu-nization coverage were sustained, weassessed compliance rates in 2009. In2009, the health centers were the pri-mary care providers for 2327 childrenaged 19 to 35 months (excludingMOGEs). Immunization coverage overallwas 88%. This 1% improvement over2008 coverage levels was borderline

significant (P = .08) (Table 3). None ofthe individual health centers experi-enced a significant decrease in immu-nization rates, and, in fact, center Bimproved by 7% (P = .02). Reexaminingthe data with all patients for whom ourcenters were the immunizers on IISrecord, in 2009, the health centerstreated 2766 children aged 19 to 35months. Overall, with this larger dataset, we could not detect a statisticallysignificant change in immunizationcoverage from 2008 to 2009 (75% vs74%, respectively, P = .44).

DISCUSSION

Our health centers implementeda comprehensive immunization bestpractices program based on an estab-lished theoretical framework. The cu-mulative effect was a 16% increase inimmunization rates overall and a 14%increase in timely immunization. Thisoccurred despite immunization rates inDC—whose overall childhood pop-ulation is similar to our patient pop-ulation, with the exception that oursincludes more racial minorities and ismore economically disadvantaged—remaining statistically unchanged (78%in 2006,20 77.6% in 2008,21 and 75.0% in20091) over the same period.

All 6 health centers improved, includingone with 3 practitioners and anotherwith .25 practitioners. Gains weresustained beyond 18 months and wereachieved by using the CoCASA soft-ware, which excluded any immunizationadministered before ACIP-recommendedminimum ages/intervals. This is a morerigid method than that used in the an-nual CDC survey, which accepts all dosesadministered.22,23 We achieved and sus-tained an 87% coverage rate despitenational shortages of PCV7 for 8 monthsin 200418 and H influenzae type b vaccinefor 18 months from 2007 to 2009.24

We implemented 6 Task Force recom-mendations and supplemented theserecommendations with an additional

QUALITY REPORT

PEDIATRICS Volume 129, Number 2, February 2012 e501 by guest on August 13, 2015pediatrics.aappublications.orgDownloaded from

activity: collaboration with communitystakeholders. According to the CCMframework, interrelationships amonghealth care institutions and the com-munity are important for improvingpatient outcomes. However, becauseprovider-community linkages requirethe cooperationof nonrelated groups, itis the least implemented of all CCMelements.25 Collaborating with the DCDOH enhanced our feedback for prac-titioners and allowed us to createpractitioner reminders. Collaboratingwith the Medicaid plans improved ouroutreach abilities.

During program implementation, allour health centers used paper records.Much time was spent reviewing chartsto determine patients’ immunizationstatus and printing IIS records for ourlarge patient panel. Since the end of thestudy, we have transitioned to an elec-tronic medical record system that hasfacilitated determination of a child’simmunizations status, because the in-formation is all in 1 location. Ideally, anelectronic record system would also

facilitate searching for unimmu-nized children and sending auto-mated phone messages to reducethe work burden of implementingclient reminders, thereby renderingactivities sustainable with lessmanpower.

WefoundthatmostWICclientsatour triallocation were already immunizationcompliant. However, ourfindingsmaynotbe generalizable, because theWIC staff atthe trial location, while not specificallychecking a child for immunization com-pliance, requested that parents bringimmunization records to every visit toforward to the IIS. Requesting immuni-zationrecordsateveryappointmentgoesbeyond what is nationally mandated,which is only to count four diphtheria-tetanus-acellular pertussis vaccina-tions by 24 months old and only atcertification visits.26 However, havingWIC personnel request immunizationrecords at every encounter may in-crease IIS completeness and remindparents of the importance of immu-nization.

Our improvement rates are consistentwith those of similar projects. Forinstance, Melinkovich et al27 in-creased immunization compliance47% among young children over an11-year span in Denver by usingprovider reminders and feedback,staff education, standing orders, andpatient reminders. Some of the gainsmay have been due to secular trends,because immunization rates improved30% in Colorado over the same timeframe.20,28 A recent Cochrane reviewfound that patient reminder systemswith and without other interventionsincreased immunization rates from1% to 20% in randomized controlledtrials.29 These studies included vari-ous combinations of patient reminders,home visits, door-to-door campaigns,patient tracking, outreach, providerprompts, and client-held records.29

Our study has limitations. We did notinclude a standard care comparisongroup. However, immunization ratesin DC overall were similar to ourbaseline rates and remained constantduring our study years.1,20,21 Somemay consider that we tested the ef-fectiveness of a comprehensive QI pro-gram to be a limitation, because wecannot gauge the relative impact ofeach intervention. However, the efficacyof each intervention has already beenwell established and summarized by theTask Force.10,11 In our estimation, and inconcordance with the CCM, it is astrength of our QI program that inter-ventions were implemented jointly, be-cause they affect different aspects of thepatient care spectrum and likely actedsynergistically.

CONCLUSIONS

We found that our pediatric health cen-ters were able to implement and sustainimmunization best practices inter-ventions over 18 months. By institu-ting these interventions as part ofa comprehensive, theory-driven, and

TABLE 3 Vaccination Coverage for 4:3:1:3:3:1:3 Vaccination Series Among Children Aged 19 to 35Months

Health Center Series Completein 2006n (%)

Series Completein 2008n (%)

P Change in CoverageFrom 2008 to 2009, %

P

A 61 (70) 105 (82) ,.05 +7 .11B 159 (74) 151 (85) ,.01 +7 .02C 866 (68) 939 (86) ,.0001 +1 .48D 168 (79) 213 (86) ,.05 21 .75E 128 (82) 205 (94) ,.001 23 .23F 37 (59) 80 (89) ,.0001 +5 .12Overall 1419 (71) 1693 (87) ,.0001 +1 .08

TABLE 4 Vaccination Coverage for 4:3:1:3:3:1:3 Vaccination Series Among Children Aged 19 to 24Months

Health Center Series Complete in 2006n (%)

Series Complete in 2008n (%)

P

A 57 (66) 94 (73) .21B 146 (68) 143 (81) ,.01C 791 (62) 855 (78) ,.0001D 156 (74) 195 (79) .18E 120 (77) 193 (89) ,.01F 26 (41) 72 (80) ,.0001Overall 1333 (65) 1552 (79) ,.0001

e502 FU et al by guest on August 13, 2015pediatrics.aappublications.orgDownloaded from

evidence-based approach to qualityimprovement, we increased immuni-zation rates .10% with a predomi-nantly minority and publicly insuredpatient population.

ACKNOWLEDGMENTSThis work was supported in part by Dis-trict of Columbia Department of Healthgrant PO310557. The Department ofHealth was involved in study design,

conduct, data collection, review, and ap-proval of themanuscript. L.Y.F. had full ac-cess to all the data in the study and takesresponsibility for the integrity of the dataand the accuracy of the data analysis.

REFERENCES

1. Centers for Disease Control and Prevention(CDC). National, state, and local area vacci-nation coverage among children aged 19-35months—United States, 2009. MMWR MorbMortal Wkly Rep. 2010;59(36):1171–1177

2. Bates AS, Fitzgerald JF, Dittus RS, Wolinsky FD.Risk factors for underimmunization in poorurban infants. JAMA. 1994;272(14):1105–1110

3. Hughart N, Strobino D, Holt E, et al. The re-lation of parent and provider characteristicsto vaccination status of children in privatepractices and managed care organizationsin Maryland. Med Care. 1999;37(1):44–55

4. Chu SY, Barker LE, Smith PJ. Racial/ethnicdisparities in preschool immunizations:United States, 1996-2001. Am J PublicHealth. 2004;94(6):973–977

5. Morrow AL, Rosenthal J, Lakkis HD, et al. Apopulation-based study of access to im-munization among urban Virginia childrenserved by public, private, and militaryhealth care systems. Pediatrics. 1998;101(2). Available at: www.pediatrics.org/cgi/content/full/101/2/e5

6. Shefer A, Smith PJ. Improving the immu-nization and health status of children inthe Women, Infants, and Children (WIC)Program. J Health Care Poor Underserved.2004;15(1):127–140

7. Smith PJ, Hoaglin DC, Battaglia MP, KhareM, Barker LE; Centers for Disease Controland Prevention. Statistical methodology ofthe National Immunization Survey, 1994-2002. Vital Health Stat 2. 2005;(138):1–55

8. Centers for Disease Control and Prevention(CDC). National, state, and urban area vac-cination coverage among children aged 19-35 months—United States, 2005. MMWRMorb Mortal Wkly Rep. 2006;55(36):988–993

9. Zell ER, Dietz V, Stevenson J, Cochi S, BruceRH. Low vaccination levels of US preschooland school-age children. Retrospectiveassessments of vaccination coverage,1991-1992. JAMA. 1994;271(11):833–839

10. Task Force on Community Preventive Serv-ices. Vaccine-preventable diseases: im-proving vaccination coverage in children,adolescents, and adults. A report on rec-ommendations from the Task Force on

Community Preventive Services. MMWRRecomm Rep. 1999;48(RR-8):1–15

11. Task Force on Community Preventive Serv-ices. Vaccinations for preventable diseases:universally recommended vaccines. TheCommunity Guide 2002. Available at: www.thecommunityguide.org/vaccines/univer-sally/index.html. Accessed July 7, 2010

12. Wagner EH, Austin BT, Von Korff M. Impro-ving outcomes in chronic illness. ManagCare Q. 1996;4(2):12–25

13. Wagner EH, Austin BT, Von Korff M. Orga-nizing care for patients with chronic ill-ness. Milbank Q. 1996;74(4):511–544

14. Coleman K, Austin BT, Brach C, Wagner EH.Evidence on the Chronic Care Model in thenew millennium. Health Aff (Millwood).2009;28(1):75–85

15. Bodenheimer T, Wagner EH, Grumbach K.Improving primary care for patients withchronic illness: the chronic care model,Part 2. JAMA. 2002;288(15):1909–1914

16. Bodenheimer T, Wagner EH, Grumbach K.Improving primary care for patients withchronic illness. JAMA. 2002;288(14):1775–1779

17. Centers for Disease Control and Pre-vention. Percentage of U.S. children .4months and ,6 years with 2+ immuniza-tions in immunization information systems(IIS), 2008. Available at: www.cdc.gov/vaccines/programs/iis/rates/2008-child-map.htm.Accessed August 16, 2011

18. Centers for Disease Control and Prevention.Notice to readers: pneumococcal conjugatevaccine shortage resolved. MMWR MorbMortal Wkly Rep. 2004;53(36):851–852

19. Centers for Disease Control and Pre-vention. National, state, and local areavaccination coverage among children aged19–35 months—United States, 2007. MMWRMorb Mortal Wkly Rep. 2008;57(35):961–966

20. Centers for Disease Control and Prevention(CDC). National, state, and local area vac-cination coverage among children aged 19-35 months—United States, 2006. MMWRMorb Mortal Wkly Rep. 2007;56(34):880–885

21. Centers for Disease Control and Prevention(CDC). National, state, and local area vacci-nation coverage among children aged 19-35

months—United States, 2008. MMWR MorbMortal Wkly Rep. 2009;58(33):921–926

22. Zell ER, Ezzati-Rice TM, Battaglia MP, WrightRA. National Immunization Survey: themethodology of a vaccination surveillancesystem. Public Health Rep. 2000;115(1):65–77

23. Smith PJ, Rao JN, Battaglia MP, Ezzati-Rice TM,Daniels D, Khare M; Centers for Disease Con-trol and Prevention/National Center for HealthStatistics. Compensating for provider non-response using response propensities to formadjustment cells: the National ImmunizationSurvey. Vital Health Stat 2. 2001;(133):1–17

24. Centers for Disease Control and Prevention(CDC). Updated recommendations for useof Haemophilus influenzae type b (Hib)vaccine: reinstatement of the booster doseat ages 12-15 months. MMWR Morb MortalWkly Rep. 2009;58(24):673–674

25. Wu S, Pearson M, Schaefer J. Assessing theimplementation of the chronic care model inquality improvement collaboratives: does base-line system support for chronic care matter?In: Luczak H, Zink KJ, eds. Human Factors inOrganizational Design and Management – VII.2003. IEA Press, Santa Monica, CA 2003: 595-601

26. National WIC-Immunization Workgroup. Im-plementing the new WIC policy for immuni-zation screening and referral: ideas andsuggestions for establishing proceduresand conducting training in your stateagency. 2002. Available at: www.nal.usda.gov/wicworks/Sharing_Center/Immunizations/Implementation_Guide.pdf. Accessed March4, 2009

27. Melinkovich P, Hammer A, Staudenmaier A,Berg M. Improving pediatric immunizationrates in a safety-net delivery system. JtComm J Qual Patient Saf. 2007;33(4):205–210

28. Centers for Disease Control and Prevention(CDC). National, state, and urban areavaccination coverage levels among chil-dren aged 19-35 months—United States,January-December 1995. MMWR MorbMortal Wkly Rep. 1997;46(8):176–182

29. Jacobson VJ, Szilagyi P. Patient reminderand patient recall systems to improve im-munization rates. Cochrane Database SystRev. 2005;(3):CD003941

QUALITY REPORT

PEDIATRICS Volume 129, Number 2, February 2012 e503 by guest on August 13, 2015pediatrics.aappublications.orgDownloaded from

DOI: 10.1542/peds.2010-3610; originally published online January 9, 2012; 2012;129;e496Pediatrics

Jacob Mbafor, Urvi Doshi and Denice Cora-BrambleLinda Y. Fu, Mark Weissman, Rosie McLaren, Cherie Thomas, Jacquelyn Campbell,

Comprehensive ApproachImproving the Quality of Immunization Delivery to an At-Risk Population: A

  

ServicesUpdated Information &

tmlhttp://pediatrics.aappublications.org/content/129/2/e496.full.hincluding high resolution figures, can be found at:

References

tml#ref-list-1http://pediatrics.aappublications.org/content/129/2/e496.full.hat:This article cites 25 articles, 2 of which can be accessed free

Citations

tml#related-urlshttp://pediatrics.aappublications.org/content/129/2/e496.full.hThis article has been cited by 2 HighWire-hosted articles:

Subspecialty Collections

munization_subhttp://pediatrics.aappublications.org/cgi/collection/vaccine:imVaccine/Immunization

_diseases_subhttp://pediatrics.aappublications.org/cgi/collection/infectiousInfectious Diseases

provement_subhttp://pediatrics.aappublications.org/cgi/collection/quality_imQuality Improvement

tion:practice_management_subhttp://pediatrics.aappublications.org/cgi/collection/administraAdministration/Practice Managementthe following collection(s):This article, along with others on similar topics, appears in

Permissions & Licensing

tmlhttp://pediatrics.aappublications.org/site/misc/Permissions.xhtables) or in its entirety can be found online at: Information about reproducing this article in parts (figures,

Reprints http://pediatrics.aappublications.org/site/misc/reprints.xhtml

Information about ordering reprints can be found online:

rights reserved. Print ISSN: 0031-4005. Online ISSN: 1098-4275.Grove Village, Illinois, 60007. Copyright © 2012 by the American Academy of Pediatrics. All and trademarked by the American Academy of Pediatrics, 141 Northwest Point Boulevard, Elkpublication, it has been published continuously since 1948. PEDIATRICS is owned, published, PEDIATRICS is the official journal of the American Academy of Pediatrics. A monthly

by guest on August 13, 2015pediatrics.aappublications.orgDownloaded from

DOI: 10.1542/peds.2010-3610; originally published online January 9, 2012; 2012;129;e496Pediatrics

Jacob Mbafor, Urvi Doshi and Denice Cora-BrambleLinda Y. Fu, Mark Weissman, Rosie McLaren, Cherie Thomas, Jacquelyn Campbell,

Comprehensive ApproachImproving the Quality of Immunization Delivery to an At-Risk Population: A

  

  http://pediatrics.aappublications.org/content/129/2/e496.full.html

located on the World Wide Web at: The online version of this article, along with updated information and services, is

 

of Pediatrics. All rights reserved. Print ISSN: 0031-4005. Online ISSN: 1098-4275.Boulevard, Elk Grove Village, Illinois, 60007. Copyright © 2012 by the American Academy published, and trademarked by the American Academy of Pediatrics, 141 Northwest Pointpublication, it has been published continuously since 1948. PEDIATRICS is owned, PEDIATRICS is the official journal of the American Academy of Pediatrics. A monthly

by guest on August 13, 2015pediatrics.aappublications.orgDownloaded from