Children and Youth Services Review 39 (2014) 160–168

Contents lists available at ScienceDirect

Children and Youth Services Review

j ourna l homepage: www.e lsev ie r .com/ locate /ch i ldyouth

Interagency Collaborative Team model for capacity building to scale-upevidence-based practice

Michael Hurlburt a,b,⁎, Gregory A. Aarons b,c, Danielle Fettes b,c, Cathleen Willging d,Lara Gunderson d, Mark J. Chaffin e

a School of Social Work, University of Southern California, Los Angeles, CA, United Statesb Child and Adolescent Services Research Center, United Statesc Department of Psychiatry, University of California, San Diego, La Jolla, CA,United Statesd Pacific Institute for Research and Evaluation, Albuquerque, NM, United Statese Department of Pediatrics, University of Oklahoma Health Sciences Center, Oklahoma City, OK, United States

⁎ Corresponding author at: School of Social Work, UnUniversity Park Campus, Los Angeles, CA 90089, United Stafax: +1 858 675 0857.

0190-7409/$ – see front matter © 2013 Elsevier Ltd. All rihttp://dx.doi.org/10.1016/j.childyouth.2013.10.005

a b s t r a c t

a r t i c l e i n f o

Available online 10 October 2013

Keywords:ImplementationSustainmentTeamsProcess modelEvidence-based practice

Background: System-wide scale up of evidence-based practice (EBP) is a complex process. Yet, few strategicapproaches exist to support EBP implementation and sustainment across a service system. Building on the Explo-ration, Preparation, Implementation, and Sustainment (EPIS) implementation framework,we developed and aretesting the Interagency Collaborative Team (ICT) process model to implement an evidence-based child neglectintervention (i.e., SafeCare®) within a large children's service system. The ICT model emphasizes the role oflocal agency collaborations in creating structural supports for successful implementation.Methods:We describe the ICT model and present preliminary qualitative results from the use of the implemen-

tation model in one large scale EBP implementation. Qualitative interviews were conducted to assess challengesin building system, organization, and home visitor collaboration and capacity to implement the EBP. Data collec-tion and analysis centered on EBP implementation issues, as well as the experiences of home visitors under theICT model.Results: Six notable issues relating to implementation process emerged from participant interviews, including:(a) initial commitment and collaboration among stakeholders, (b) leadership, (c) communication, (d) practicefitwith local context, (e) ongoing negotiation and problemsolving, and (f) early successes. These issues highlightstrengths and areas for development in the ICT model.Conclusions: Use of the ICT model led to sustained and widespread use of SafeCare in one large county. Althoughsome aspects of the implementationmodel may benefit from enhancement, qualitative findings suggest that theICT process generates strong structural supports for implementation and creates conditions in which tensionsbetween EBP structure and local contextual variations can be resolved in ways that support the expansion andmaintenance of an EBP while preserving potential for public health benefit.

© 2013 Elsevier Ltd. All rights reserved.

1. Introduction

Introduction of evidence-based practices (EBPs) can lead to substan-tial public health benefits. However, the implementation process canshape whether intended outcomes are actually achieved (Aarons &Palinkas, 2007; Allen, Brownson, Duggan, Stamatakis, & Erwin, 2012;Crea, Crampton, Abramson-Madden, & Usher, 2008; Fixsen, Naoon,Blase, Friedman, & Wallace, 2005; Greenhalgh, Robert, Macfarlane,Bate, & Kyriakidou, 2004; Palinkas & Aarons, 2009). Well-establishedpractice models, implemented poorly or not sustained, will fail toachieve intended goals despite research evidence supporting their

iversity of Southern California,tes. Tel.: +1 858 675 0167x253;

ghts reserved.

clinical effectiveness (Backer, 2000; Bond, Drake, McHugo, Rapp, &Whitley, 2009). Thus, an effective implementation approach is often asimportant as the practice to be utilized.

Several conceptualmodels describe factors that can influence imple-mentation effectiveness. Some models emphasize structural featureshypothesized to be core components of effective implementation(Aarons, Hurlburt, & Horwitz, 2011; Damschroder et al., 2009;Feldstein & Glasgow, 2008; Greenhalgh et al., 2004; Mendel, Meredith,Schoenbaum, Sherbourne, & Wells, 2008). Other models emphasizeimplementation processes, outlining key steps (and their timing) hy-pothesized to contribute to successful implementation of service inno-vations (Glisson & Schoenwald, 2005; Sosna & Marsenich, 2006;Stetler, McQueen, Demakis, & Mittman, 2008). Structural and processimplementation models are often conceptually aligned. For example,both types of models address the central importance of issues such as

161M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

strong and effective leadership to support change initiatives, establish-ing a strong fit between change efforts and organizational and servicesystem culture and values, creating methods for ensuring quality pro-gramdelivery (i.e.,fidelity), and clarifying/addressingfinancial supportsfor a change initiative.

This paper describes the Interagency Collaborative Team (ICT) im-plementation process model. The ICT model provides an approach tosupport successful roll-out of human service innovations in large geo-graphic areas, particularly change efforts involving EBPs. It is directlyrelevant to improving outcomes of service enhancements in child andfamily service systems. The ICT model is designed to enable organiza-tions to work together in ways that generate the structural and processsupports associatedwith successful implementation and sustainment ofinnovations.We discuss some core areas of difference and similarity be-tween the ICT model and other implementation strategies, connectingcore features to one structural implementation framework, the Explora-tion, Preparation, Implementation, and Sustainment (EPIS) framework(Aarons et al., 2011). Qualitative data from the scale-up of an EBP inone large county illustrate areas of strength and some limitations inthe ICT model and provide perspective on other process models of EBPimplementation.

1.1. Interagency Collaborative Team (ICT) model in the EPIS Framework

Like a number of implementation frameworks, the EPIS frameworksummarizes variables that can positively or negatively affect the imple-mentation of an evidence-based practice. The EPIS framework is unusualin identifying key variables thought to particularly affect implementa-tion efforts during each of four major implementation stages in publicsector child welfare and mental health settings. For example, some keyvariables identified as influencing the preparation and early implemen-tation stages of a quality improvement effort include strength of theleadership supporting change (Aarons, 2006; Edmondson, 2004; Klein,Conn, & Sorra, 2001), the degree of fit of an innovation with the servicesystem context (Klein & Sorra, 1996), clarity of financial support forproposed changes (Aarons, Wells, Zagursky, Fettes, & Palinkas, 2009;Frambach & Schillewaert, 2002), level of involvement of practice devel-opers in the implementation process (Aarons et al., 2011), and the pres-ence of cross-organizational knowledge of and commitment to the newpractice (Glisson & Schoenwald, 2005; Sosna & Marsenich, 2006).

The ICT implementation process model outlines steps designed tolead directly to the kinds of key implementation supports described inthe EPIS framework. The model takes its name from the fact that itemphasizes the key role of collaboration among stakeholders and staffmembers at the system level, from multiple partnering organizations,and of developing or utilizing a local “seed” team to embody and sup-port promotion and maintenance of expertise and ongoing fidelity inthe practice to be implemented. Inter-agency collaboration andwilling-ness to share expertise is central to multiple steps in the implementa-tion process and across organizational levels. Conceptually, the ICTmodel hasmuch in commonwith other implementation processmodels(Chamberlain, Price, Reid, & Landsverk, 2008; Glisson & Schoenwald,2005; Sosna & Marsenich, 2006), which describe logically ordered setsof activities designed to create a context in which EBP implementationoccurs effectively and intended public health benefits are realized.

1.1.1. ICT processes and action stepsFig. 1 provides a graphical representation of key implementation

processes included in the ICT model, with the stages of the EPIS frame-work listed temporally down the left side of the figure. In the ICTmodel,a process is considered to be a goal-driven domain of focus that extendsover a period of timewithin the longer implementation effort. For exam-ple, the initial EBP education and stakeholder development and align-ment processes involve an initial phase of identifying community-based stakeholders with interests in a particular practice change effort,and discussions and education efforts designed to lead to joint selection

of and commitment to a common practice change initiative. The practicefit assessment process involves a careful analysis by key stakeholders atsystem and organizational levels of EBPs under consideration to identifyaspects of practices that fit with existing policies, contracting, and ser-vice routines and thosewheremodificationsmight be required. Brief de-scriptions of each ICT process are provided at the bottom of Fig. 1.

Specific ICT model action steps are listed in Table 1 that animate theprocesses shown in Fig. 1. Their contributions to each implementationprocess are noted in the figure. For example, the Initial EBP Educationprocess occurs as part of ICT action steps A (convening of stakeholders)and B (soliciting expertise). Education about the EBP becomes an in-tense process focus that occurs in the context of meetings among inter-ested stakeholders, supplemented by expertise about the EBP solicitedfrom appropriate sources. Sources may be multiple, including EBPdevelopers, other users of the EBP, researchers having familiarity withthe practice, and/or materials available from sources such as journalsor intermediary organizations that summarize information aboutEBPs. Structural supports designed to arise from the ICT processes arerepresented as planks beneath the model processes that generatethem.We represent the ICTmodel in thismanner because it is best con-ceptualized as a series of major actions that address core implementa-tion processes. Specific action steps animate these processes and giverise to or strengthen key structural supports viewed as creating an envi-ronment that can sustain an innovative practice as it is scaled up.

1.1.2. ICT initial steps: exploration/adoption decisionThe ICT model initially revolves around a service system and multi-

agency commitment to invest in the long-term viability of an EBP-centered quality improvement initiative, with an ultimate goal to im-prove selected client level outcomes. Partnering agencies may includea range of stakeholder organizations, but particularly involve funding,administration, and service delivery organizations from the outset. Dur-ing an initial exploration phase, stakeholders convene and meet to dis-cuss need for a practice change effort that involves investment bymultiple individuals and organizations. Although no specified leader isrequired to initiate suchmeetings, it is expected that one local or region-al organization will often take responsibility for convening and leadingsuch discussions. For example, a health and human service administra-tion may convene discussions around maltreatment prevention, reduc-tion in delinquency, or some other practice change effort. Within anICTmodel-guided implementation, convening of stakeholders should in-clude efforts to identify those stakeholders with substantial interests inthe identified substantive area (e.g., child neglect).

A second important step in the process of exploring a possible prac-tice change involves concentrated efforts to obtainwide-ranging factualinformation about the costs, benefits, and tradeoffs associatedwith spe-cific practice changes. Outside expertise is identified and sought to helpanswer questions and reduce uncertainty about the change effort underdiscussion. The joint process of participating in education about possiblepractice change efforts and discussing the advantages and disadvan-tages of various options is aimed at developing a shared commitmentand direction among stakeholders at an inter-agency level to a jointlysupported EBP implementation.

1.1.3. Interagency seed teamdevelopment: preparation and implementationOnce a specific EBP is selected as the focal point for a broad practice

change effort, stakeholders in the ICT process initiate implementation ofthe EBP by creating a formative interagency collaborative “seed” team(or ICST), which may consist of employees from several different localorganizations that form a core unit of expertise in the selected servicemodel. A seed team intentionally involves multiple organizations inthe maintenance of innovation expertise to build broader investmentin, commitment to, and communication about an innovation amonginvested stakeholders and subsequently trained practitioners. The seedteam becomes a repository of local expertise for an EBP. It is designedto serve as the ongoing support structure for continued EBP training,

Fig. 1. Implementation processes emphasized by the ICT, including structural supports hypothesized to emerge from following ICT model steps listed in Table 1.

162 M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

coaching, and roll-out across a geographic area and as a facilitator ofminor practice adjustments that help to fit a practice to diverse regionalcontexts (Aarons et al., 2012). Members of the seed team maintain acentral liaison role between the EBP developer and other actors atmultiple levels within the service system so that issues, decisions, andadaptations can be negotiated during initial experimentation withimplementation.

1.1.4. Seed team: ongoing responsibilitiesFollowing inception of the seed team and initial EBP training, certifi-

cation, and service delivery, the team is then responsible for trainingand supporting additional teams of individuals that can then implementthe selected EBP as it is scaled up across a service system. The seed teamassumes responsibility for ongoing training of new teams of

practitioners that may consist of employees from several local non-profit organizations, hereafter referred to as Interagency CollaborativeTeams (ICTs). These ICTs form for the express purpose of learning andmastering delivery of the EBP to be implemented, under the guidanceof the original seed team. Althoughmembers include staff frommultipleorganizations, they meet together with a seed team coach during a su-pervision and knowledge transfer phase. ICTs trained by the seedteam are responsible for the primary delivery of the EBP. This imple-mentation structure, with regional teams having interagency composi-tion, results in a network of local providers that allows for high inter-agency communication, and information and possible workloadsharing.

The seed team maintains relationships with ICTs following EBPtraining. In order to maintain and continually enhance quality delivery

Table 1Steps involved in the Interagency Collaborative Team (ICT) implementation processmodel.

ICT steps

A. Identify and convene stakeholders with likely interests in a shared qualityimprovement initiative (may be iterative)

B. Solicit relevant expertise required to address questions about selected qualityimprovement directions and EBP alternatives

C. Develop commitment and direction among stakeholders to a jointly supported,EBP-centered change effort

D. Create an interagency seed team to:1. Learn the EBP2. Conduct initial local delivery of the EBP3. Train new local EBP practitioners4. Serve in a liaison role with external EBP developers/trainers5. Monitor and provide feedback about quality of EBP delivery6. Communicate and support a commitment to quality EBP delivery7. Communicate with stakeholders about implementation progress

E. Form additional interagency training teams that:1. Deliver the EBP2. Relay feedback about implementation to the seed team3. Share information with one another about implementation progress

F. Plan a phased reduction in EBP developer involvement

163M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

of the EBP, newly trained providers continue to receive constructivesupport and feedback in the form of supervision and coaching fromseed team members for a defined period of time, which may vary bypractice or situation. In the case of SafeCare, ongoing fidelitymonitoringand coaching are integral components of the EBP. Organizing supervi-sion and coaching through the seed team has many potential benefitsin the short and longer term. This structure is designed to provide aclear source of leadership and information to newly trained providers.Ideally having the seed team serve as the source of ongoing trainingand coaching facilitates a gradual reduction of EBP developer involve-ment whereby the local service system and its contracted agencies be-come the repository of expertise in the practice being adopted. Basedon its initial key role within a multi-organizational implementation ef-fort, the seed team continues to serve as a locus of information aboutneeds for adaptations to make a practice work within a particular localcontext. Such adaptations may involve changes to aspects of the inter-vention itself or to the structure of the service setting in which theEBP is delivered (e.g., Finno-Velasquez, Fettes, Aarons, & Hurlburt,under review). Cross-organizationalmembership on the seed teamcon-tributes to ensuring a continuing locus of expertise available to all orga-nizations within the ICT partnership, reducing the kinds of expertiseloss that regularly occur within individual organizations and agenciesdue to staff turnover and organizational changes.

As noted above, the ICT model seeks to foster implementation sup-ports in the areas of practice fit, leadership, communication, expertisedistribution, EBP quality (fidelity) management, effective developer in-volvement, and program adaptation. A large California county used theICTmodel to implement an evidence-basedneglect prevention programthroughout the county. Qualitative inquiry into this effort helps to illus-trate several key elements of the implementation approach.

1.2. The present study

Implementation of the SafeCare neglect prevention model occurredin one large California county. With approximately 3million total resi-dents, the county population is similar in size or larger than that ofmany smaller U.S. states. The county encompasses urban, semi-urban,and rural areas that are home to a diverse cultural mix of residents, in-cluding significant Mexican-American and Native American popula-tions. Planning for many aspects of public human services in thecounty is organized into local planning regions, each with some of itsown local history, demographic and cultural characteristics, climate,and topography. Implementation of any new practice at a county level

represents a large-scale system and organizational change effort thatoccurs across the planning regions.

In 2007, the Department of Health and Human Services (DHHS)agency and the local chapter of a national foundation embarked uponan effort to transition one category of county maltreatment preventionservices toward an EBP. County DHHS leaders (responsible for childwel-fare services), members of the local branch of a national foundation, andresearch partners convened to consider three different child focusedEBPs to improve outcomes for children and families involved with thechildwelfare system. After consideration of research evidence, program-matic fit, and financial resources required, SafeCare®, an evidence-basedchild neglect prevention programutilizing homevisiting (Chaffin, Hecht,Bard, Silovsky, & Beasley, 2012; Lutzker, Bigelow, Doctor, Gershater, &Greene, 1998), was selected for implementation.

Qualitative interviews and focus groups allowed us to document theroll-out of SafeCare, provided insight into how the ICT model generatedkey structural supports for implementation, and helped to identifyprocess issues worthy of more careful consideration. The following sec-tion summarizes at a general level what we learned from that qualita-tive work about themes related to implementation process, including:(a) initial commitment and collaboration among stakeholders, (b) lead-ership, (c) communication, (d) practice fit with local context, (e) ongo-ing negotiation and problem solving, and (f) early successes.

2. Methods

2.1. Overview

From August, 2008 to January, 2009we undertook in-depth qualita-tive interviews with key stakeholders involved in the early stages ofsystem-wide implementation of SafeCare that followed the ICT imple-mentation model. Data collection and informed consent procedureswere approved by the appropriate Institutional Review Boards.

2.2. Participants

Participants in this study included 27 stakeholders involved in vari-ous facets of the early implementation process. Participants were re-cruited through an initial telephone call or email describing the studypurpose and participation. One of the authors either made the initialcontact or was available to answer questions about participation. Ourpurposive sample consisted of all individuals who took part in initialEBP planning meetings, including representatives from the county(n = 3) a foundation supporting part of the implementation effort(n=9), and the executive directors of the community-based organiza-tions (n=3) that were eventually contracted to deliver SafeCare. Next,we interviewed key individuals involved in supporting delivery of theEBP, including SafeCare supervisors, trainers, and coaches (n=6) andfront line providers (n=6).

2.3. Semi-structured interviews

Interview guides consisted of open-ended questions that were tai-lored to each stakeholder group. The interviews with representativesof the county, the foundation, and community-based organizations fo-cused on the initial planning process, their roles and responsibilitiesand interactions with one another, and perceptions overall of SafeCareimplementation. These interviews also sought to capture data onorganizational- and system-level factors affecting implementation. Theinterviews with supervisors, trainers, coaches, and seed teammemberscentered on each person's involvement in the ICT approach, knowledgeof and experiences with SafeCare, and the “fit” of the intervention withlocal populations and service delivery contexts. All participants agreedto recorded interviews, which lasted approximately 60min, and wereprofessionally transcribed. Transcriptions were reviewed for accuracy

164 M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

by a research assistant. In addition, ethnographer interview notes weretyped and uploaded to an electronic database.

2.4. Data analysis

Weemployed an iterative process to review the textual data from in-terviews and utilized NVivo 9 (2009) qualitative data analysis softwareto facilitate this work. Data analysis proceeded first by engaging in anopen coding approach to locate the themes and issues that emergedfrom the interview transcripts. Focused coding was then used to deter-mine which of these themes emerged frequently and which represent-ed unusual or particular concerns (Emerson, Fretz, & Shaw, 1995). Inthis staged approach to analysis, our research team coded sets of tran-scripts, created detailed memos linking codes to each theme and issue,and then passed their work to other teammembers for review. Discrep-ancies in coding and analysis were identified and resolved through con-sensus during team meetings (Sandelowski & Barroso, 2003). Themesemerging from analyses were cross-walked with objectives of the ICTmodel to identify areas in which language used by participants mightdiffer from that used in the model but have similar meaning. Principalthemes from the interviews are presented, supplemented by relevantcomments fromparticipants. In some cases, quotations are edited slight-ly to smooth readability.

3. Results

Results confirmed the significance of a number of the processes andstructural supports outlined in the ICT model, including initial commit-ment and collaboration, cross-level leadership, and practice fit to thelocal context. Other key themes also arose, such as the importance ofearly successes, and of negotiating roles and responsibilities amongstakeholder organizations. Interconnections among these themes arehighlighted below.

3.1. Commitment and collaboration

The first step in implementing the ICT model involves identifyingand then bringing together key stakeholders invested in a change ef-fort in order to discuss shared interests in undertaking a particularinitiative. In this case, interest from stakeholders in the child welfaresystem and the foundation chapter in supporting a system-wide im-provement in the area of child maltreatment led to the creation of a“Council” focused on a possible quality/capacity enhancement effortthat eventually centered on SafeCare. The Council included represen-tatives not simply from county child welfare services and the localfoundation, but from community-based non-profit organizations,advocacy organizations, a children's hospital, EBP developers, andresearchers. Initial conversations about directions for the interventionwere critical in setting the foundation for the long-term collaborationamong stakeholders that would be required to implement and sustainSafeCare.

A potentially underappreciated aspect of building initial commit-ment is the fact that stakeholders do not necessarily share the sameorganizational culture or values when they begin to collaborate. Inthe case of SafeCare, some stakeholders worked in large governmentorganizations, others in small non-profit organizations. Consequent-ly, stakeholders often had very different ideas about how to pursuechange and their respective roles in this process, which at times ledto tension and conflict that added complexity to the implementation,some examples of which are described in a later section. Nonethe-less, the stakeholders pushed forward, buoyed by the belief thatthe specific EBP they had agreed to implement would improvechild welfare services and reduce neglect. County officials, for exam-ple, were enthusiastic about ensuring that services provided in thechild welfare system were evidence-based. Their confidence in mov-ing forward with SafeCare was bolstered after a presentation and

discussion period with a research team knowledgeable about the EBP,its underlying evidence base, and the benefits of the intervention.

The initial commitment process involved frank discussion and eval-uation of whether interests were broadly shared among stakeholdersamidst differences in organizational directions, cultures, and values.Although this process of group reflection may not have been sufficientto guarantee the overall success of the implementation effort, it did ap-pear to have been a central component in laying an appropriate founda-tion for positive outcomes. This process put the diverse interests of thestakeholders on the table, fostered commitment to a common direction,and engendered a sense of top-level leadership support for the changeeffort that was ultimately reinforced by additional leadership tiers.

3.2. Leadership

Once an intervention is selected, the ICT model calls for develop-ment of a “seed team,” an initial cadre of service providers responsiblefor acquiring expertise in the service model, for transmitting thisknowledge to other teams of individuals involved in day-to-day serviceprovision, and for providing ongoing fidelity assessment and support.By virtue of their roles as trainers and supervisors of future cohorts ofSafeCare-trained home visitors, the nine seed team members wereplaced in a structural position of leadership. As the implementationprogressed, three of the original seed team members were selected toassume the roles of trainers/coaches, and one emerged as the teamleader and SafeCare supervisor. As noted by several participants, theseindividuals could be counted on to guide newly trained home visitorsin consistent SafeCare practice and thus were paramount to implemen-tation success.

As noted, one SafeCare supervisor became the primary identifiedteam leader and source of support for home visitors. Several home vis-itors commented on the support provided by the SafeCare supervisor inparticular. One home visitor stated, “She's very good at answering ourquestions”, while a second added, “I find her very helpful and she'savailable if I need her.” Although not expressed in terms of leadershipfrom the home visitor perspective, we interpreted the regular com-ments regarding the support and information provided by the SafeCaresupervisor as a reflection of the clinical leadership provided by the seedteam, as viewed by home visitors.

Strong leadership was also evident from the directors of eachcommunity-based provider organization, the local foundation, andcounty child welfare services. The fact that the provider organizationscollaborated from the outset to respond to the local foundation's Re-quest for Proposals to deliver SafeCare from a multi-agency position,and then facilitated involvement of their staff within a single seedteam, reinforced a broad sense of cross-level leadership commitmentto SafeCare. During the implementation phase, the local foundation, inpartnership with the county, also spearheaded organizational meetingsfor planning purposes. This higher-level buy-in, commitment, and sup-port communicated a message that this new EBP was not the “flavor ofthe day” and that therewas an expectation for effective implementationand ongoing use of SafeCare.

3.3. Communication

The majority of participants suggested that communication wascrucial to successful implementation, but attributed problems en-countered during the roll out of SafeCare to communication chal-lenges. Interviewees reported that the communication structuresaround implementation were initially insufficient. Some stakeholderswere privy to misinformation or to no information regarding issuesimpacting implementation. In one example, a county staff member in-correctly informed some supervisors and the local foundation thathome visitation caseloads pre-SafeCare were half of what they were inreality (typically 20 vs. 10 cases per home visitor). Such unintentionalmisinformation altered the course of project planning and fueled

165M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

concerns that implementation of the newEBPwould prove to be too ex-pensive in the long run. Problems disseminating information amongstakeholders across all levels were also common. Email, in particular,did not function as a dependablemode of communication and informa-tion sharing. There were times when stakeholders felt that only a limit-ed subset of individuals were receiving needed information aboutprogram implementation.

Stakeholders also described discussing or even deliberating onSafeCare issues individually outside of group meetings, while alludingto underlying power dynamics that influenced communication. Oneparticipant lamented about being left of out of these informal communi-cations about SafeCare: “When I hear [other stakeholders] talking toeach other, I can tell that [they] have talked on the side…. It looks likethey're able to access the information they need, and [can] find thingsout.” A second participant reported running into others involved in im-plementation in non-SafeCaremilieus and felt freer to share ideas aboutthe intervention and its roll out. They admitted, “We talk about[SafeCare] a bit. [If] I am with the county person, I'm like, ‘Don't askme [later at a Council meeting], I'll talk to you here. I won't talk to youin the meetings when there are other county people [present]’.” Suchcomments suggest that perceived power imbalances among stake-holders sometimes interfered with candid dialog.

Structurally, the SafeCare supervisorwas expected to serve as a locusof communication and information exchange between home visitorsnewly trained in SafeCare and the other stakeholder groups (e.g., countychild welfare and the local foundation). The SafeCare supervisor com-mented on the challenges she experienced mediating between thehome visitors and upper-level leadership: “Right now…we're in theearly stages [of implementation] so there's a lot of things that are chang-ing everyday…. [There's a need to] maintain that open communica-tion…. That's really the big part of it, as far as ‘we want you guys to dothis’ or ‘we want you guys to do this differently’ or ‘don't use thisform’ or ‘use this form.’ I tell the team, ‘There's always going to besome changes and as soon as I know something you guys will know aswell. So just bear with me. The first few months are going be likethis.’” In some respects it was difficult for the SafeCare supervisor andthe home visitors to keep up with frequent changes initiated fromabove, due to the potential lack of clearly structured communicationchannels. Apart from the SafeCare supervisor, the home visitors alsostruggled to some degree with whom else they should be communicat-ing regarding SafeCare implementation. In particular, they were oftenunsure whether they should turn to or report to their immediate teamsupervisor, who was not trained in SafeCare, but who managed theirworkaday lives, or to others involved in the roll-out, such as the re-searchers or trainers.

3.4. Fit with existing practice and fidelity

The ICT model created an active, functional process for addressingthe fit of SafeCare to the structure, culture, and local needs of servicepopulations and the organizations delivering those services. This oc-curred at several points. During the initial discussion and commitmentphase, possible EBPs were considered with respect to their target audi-ences and outcomes, modes of delivery, and training and resource re-quirements. SafeCare emerged from this phase as a top candidate. Ithad a jointly held focus of interest to Council members (child neglect),only involved retooling of the curricular component of existing homevisitation services rather than a more substantial reformulation of ser-vice models, and had implementation costs viewed as manageablewithin a large-scale roll out of the practice. This first participatoryphase put stakeholders on the path to selecting a model perceived tohave good fit with organizational structure, values and needs.

At the practice level, the structure of the seed team created an idealframework for adapting an EBP as it was progressively implementedacross a larger scale service area. A major theme, widely expressed inour interviews, was concern for whether SafeCare could be applied to

all families. The seed team described the service population as havingmultiple needs, not necessarily consistentwith the SafeCare curriculum.Families, for example, struggled with serious problems, ranging fromthe procurement of basic necessities to shelter, critical issues thatfell outside the scope of SafeCare. Commenting on the difficulties of car-rying out SafeCare in such circumstances, one seed team member ex-plained, “[Families] are having a hard time. How can somebody want tolearn about safety or health when they don't know what they're goingto feed their kids [or] have the money to pay their rent?” This individualadmitted to cutting short her visits with families in order “to help themwith resources or whatever they're going through.” One solution ad-vanced during the early implementation periodwas not to begin SafeCarewhen families were in crisis. In such cases, the seed team member onlyinitiated SafeCare after he or she had the opportunity to help the familyaddress other issues. Over time, the seed team helped to shift home visi-tation practice so that such issues could occur within the context of ongo-ing SafeCare visits. In fact, such issues were fit within the broaderproblem-solving framework utilized within SafeCare. The seed teamplayed an instrumental role in identifying this issue and facilitating incre-mental solutions that helped tailor SafeCare to the local context.

The seed team was also able to undertake deliberative steps to im-prove the cultural responsiveness of SafeCare to populations in thelocal region. For example, the seed team actively reviewed and translat-ed SafeCare materials to improve their relevance to Latino, particularlyMexican-American, families. The seed team also discussed and adaptedchild health focused sessions andmaterials to the customs of Latino im-migrant families, who sometimes expressed values and preferences forhomeopathic remedies not originally considered within the healthmodule of the SafeCare EBP (Finno-Velasquez et al., under review).

The ICT model requires that the seed team become the local cross-agency repository of expertise in an EBP for a group of collaborating or-ganizations. At the outset, this involved the seed team learning and uti-lizing the new practice model until they reached a level of expertisemeriting certification by the model developers. While mastering theEBP during early service delivery, the seed team was also immersed inthe delivery of the practice in the local context. In their designatedroles as future trainers, coaches, and supervisors, the seed team mem-bers were tasked with internalizing the knowledge of and expectationsof program developers, but also with a high level of responsiveness tolocal population needs. As noted above, the pressure to resolve tensionsbetween existing SafeCare structure and knowledge of the local servicepopulation put the seed team in the position of developing a locally re-fined expertise that fit the practice to the local area and allowed for aplanned decrease in the involvement of the original EBP developer insupporting sustainment of SafeCare. Documentation of the specific ad-aptations made by the seed team is discussed by Finno-Velasquezet al. (under review). Participants viewed the gradual decrease in devel-oper involvement as proceeding effectively, as originally planned.

3.5. Negotiation of rights, roles, responsibilities, and interests

In addition to a need to incorporate further structure in the area ofcommunication patterns, as alluded to by earlier qualitative findings,the ICTmodelmight benefit from further attention to enablingmethodsfor facilitating negotiation of differences among organizations and theirmembers. Although many possible differences among partners mayemerge during the implementation of any new change effort, severalspecific examples from the SafeCre implementation experience illus-trate the kinds of issues that arise and require negotiation during alarge-scale collaborative EBP initiative. For example, the appropriatepace for SafeCare implementation and documentable change was onearea in which stakeholders held differing expectations and opinions. Itarose because the collaborating partners had different needs and expec-tations influencing their participation. The local foundation supportinginitial training and development of the seed teamhad interests in seeingmeasurable outcomes quickly in order to show progress to donors

166 M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

supporting the organization, in addition to their core interest in improv-ing outcomes for children and families. Other stakeholders did not sharethe pressure to accelerate change to the samedegree. Discordant expec-tations led to some tensions related to this issue.

Differences also emerged with regard to expected roles and respon-sibilities. For example, one important decision during implementationrevolved around the order in which the seed team trained new homevisitor teams in the various county regions. As a new partnership inwhich different parties jointly contributed to the SafeCare implementa-tion, issues around rights and responsibilities needed to be negotiated.County representatives had the authority and perceived responsibilityto make decisions about ordering of SafeCare roll-out, since they wereentrusted by the public to provide oversight of child welfare services.However, foundation staff also sought to exert authority in this area be-cause the foundation had committed funds to support the seed team. Asshared authority was being established, power struggles betweenparties occasionally ensued. These tensions were recognized by othersinvolved in the implementation. One community-based provider orga-nization executive stated, “It's no one's fault, but I think the foundationand the county still have to have some meetings about whose role iswhat, and who has decision-making authority on certain things. And Ithink right now they are still a little messy.”

One further example may also be illustrative. Having significant in-vestment in the success of the SafeCare implementation, especiallygiven that thiswas thefirst time the foundationwas investing in a singlemajor capital improvement as opposed to multiple smaller local grants,foundation representatives indicated a need and a right towork directlyat times with the local community-based organizations contracted bythe county to deliver SafeCare services. County representatives, againhaving direct responsibility formanagement and oversight of childwel-fare services, also asserted a duty to be involved in conversations andcommunications around service initiatives under their purview, some-times leading to tensions around roles and responsibilities amongcollaboration partners. One participant described the resulting tensionas a “strange triangulation” between the entities involved. However,“What I think helped move it [the intervention] along was the enthusi-asm of everyone on the Council, [and] the enthusiasm of the County, fortransitioning into the SafeCare model.”

3.6. Early successes

The ICT model intentionally involves the staged deployment of anEBP, beginning with the experience of the seed team prior to subse-quent roll out to and support of other practitioners. Participants in qual-itative interviews consistently mentioned that the purposeful transitionof SafeCare from an initial seed team did lead to early signs of successthat were interpreted as facilitating and supporting efforts to imple-ment and sustain SafeCare. One participant noted that stakeholdersacross all levels were “…seeing it is working. They are seeing thatthey're not getting cheated or embarrassed. They are seeing that thereis a system in place. They're just calmer. And so their control needs area little less.” Markers of success appeared to help stakeholders beginto overcome power struggles. Evidence of success took various forms,including positive experiences with initial training and delivery of ser-vices and encouraging reports by the home visitors about how familieswere responding to the intervention. A county official noted, “Thetrainingwentwell and implementationwas successful…. The receptionactually at the line level has exceededmy expectations…. Home visitorsare comfortable with one, embracing more of a script and, you know,more structure within the visit and that they're receptive to thecoaching…” Undertaking implementation in a phased roll out, begin-ning with exploration, appropriate preparation, and planned imple-mentation phases and involving the seed team as a central trainingand support component, created opportunities for shared successesand further commitment to ongoing problem solving.

4. Discussion

This paper describes the ICT model for EBP implementation, whichis designed to facilitate development of many supports hypothesizedto be central to successful quality improvement efforts organizedaround EBP implementation. Qualitative data from implementation ofSafeCare in one large geographic area provided the opportunity toreflect on the strengths and limits of the ICT model and to consider itrelative to other process models of quality improvement and EBPimplementation.

The ICTmodel departs from traditional service structure and processby distributing local expertise across service teams, and more focallyin a seed team, in a way that takes into account challenges faced byreal-world public social service systems (Aarons et al., 2011; Aarons,Sommerfeld, Hecht, Silovsky, & Chaffin, 2009). Drawing on computerscience and engineering theories of distributed expert systems (Dai,Xie, Poh, & Liu, 2003) and team decision making (Hollenbeck et al.,1995), the ICT model aims to increase effective team functioningthrough building greater systemic and cross agency trust (Edmondson& Roloff, 2009) and collaboration (Bertram, 2008). This structuring oc-curs at multiple levels, including among administrative and fundingstakeholders with interest in a practice change initiative, and at thelevel of local clinical leadership (i.e., the seed team). The goal of theICT process is to build interagency relationships at both levels, and be-tween levels, creating the structural supports central to effective adop-tion, implementation, and sustainment of an EBP with positive publichealth effects. With the seed team playing a substantial role in opera-tional implementation, the ICTmodel seeks to build structures and pro-cesses that enable the fitting of an EBP to the local context as outsidedeveloper involvement is reduced, and potential for EBP sustainmentis increased.

Other implementation strategies directly relevant to EBP implemen-tation include such models as the Availability, Responsiveness andContinuity (ARC) and Community Development Team (CDT) models.Each arises out of somewhat different theoretical frameworks than theICT model. The ARC model emerged from organizational development(Burke, 1993; Nadler & Tushman, 1977; Porras & Robertson, 1992)and interorganizational domain development (Gray, 1990; Trist, 1985)theories. The ARC model focuses strongly on improving organizationalculture and climate and organizational processes to support effectivecare andmore effective EBP implementation. ARC relies on an organiza-tional change agent to work with the organization to effect intra-organizational change to improve care. The CDT model was developedbased on the experiences of the California Institute for Mental Health(CiMH), a training and technical assistance organization supported bycounty mental health agencies and child welfare systems. The CDTapproach focuses on developing supportive collaborations amongstakeholders, often in different counties, that are considering andimplementing EBPs. The CDT model, like the ARC, relies heavily on theinvolvement of an outside consultant to structure communicationsamong stakeholders, set priority topics for discussion, and foster organi-zational problem solving around issues that arise during implementa-tion. The outside consultant brings collected expertise acquired fromextensive communications with multiple EBP program developers andlocal stakeholders to his/her role in supporting EBP implementation ef-forts (Sosna & Marsenich, 2006).

Considerable common ground exists among these models, althoughthere are also some areas of difference, both qualitatively and in overallemphasis. For example, the ICT model devotes considerably less atten-tion to efforts to change intraorganizational culture and climate thanthe ARC model. However, both have many similarities including pro-cesses specifically targeting initial collaborative work and joint decisionmaking among stakeholders, and the development of mechanisms formonitoring and providing feedback about implementation progressand quality, and for fitting practices to be implemented to the local geo-graphical and cultural context. The ICT and CDT process models also

167M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

share similarities. Both include processes specifically targeting initial col-laborative work among adopting stakeholders and organizations, usingeducation to reduce uncertainty about EBP adoption during the explora-tion and preparation stages, and developing a core focus on EBP fidelityas part of implementation and sustainment. The models differ, however,in howmany of the implementation processes are organized by an exter-nal change agent as opposed to individuals within the service system.Both the ARC and CDT models include a much more extensive role foran external change agent than the ICT model, which proposes that theplanned actions and processes illustrated in Fig. 1 will result in the devel-opment of structural supports at the core of effective implementation. TheICT model relies more on establishing a process map for developing localstructures to support implementation, and on facilitating desired inter-agency relationships, than on external consultant support.

Given notable common procedural aspects of different implementa-tion process models, the rich qualitative data gathered around imple-mentation of SafeCare in this study informs the ICT model and, tosome degree, other implementation process models as well. One keyfinding from this qualitative study was the interconnectedness of theprocess components and the implementation supports they facilitated.Initial collaborative efforts among stakeholders, including recognitionand discussion of differences, facilitated later problem solving and ne-gotiation around areas of disagreement and potential conflict. Stagedroll out of implementation created opportunities for early success,which in turn facilitated communication and problem resolution andhelped build “buy-in” and enthusiasm for the EBP. The presence of theseed team solidified perceptions of leadership at multiple levels and di-rectly addressed other key implementation drivers, including sustain-ing a focus on fidelity and fitting of the practice to be implemented tokey local contextual variations. Our qualitative work supports argu-mentsmade by others that multi-component approaches to implemen-tation that address inter- and intra-organizational contextual issues arenecessary to create an environment conducive to strong implementa-tion and sustainment (Aarons et al., 2011; Damschroder et al., 2009;Ferlie & Shortell, 2001; Fixsen, Blase, Naoom, & Wallace, 2009; Glisson& Schoenwald, 2005; Grimshaw et al., 2001; Grol & Grimshaw, 1999).Results from interviews clearly connected various aspects of the imple-mentation process to the overall supports for implementation. The in-terviews also are consistent with the idea that core implementationdrivers reinforce and support one another (Fixsen et al., 2009).

The ICT model includes notably less involvement of externalchange agents, such as intermediary or brokering organizations, oroutside organizational development consultants than other processmodels. The foundation and its willingness to support initial imple-mentation phases were important to SafeCare implementation be-cause the foundation served as the initial convening agent and wasresponsible for providing financial support to initiate implementa-tion of SafeCare. However, the ICT model relies more heavily on orga-nizing a series of actions and local structures designed to createinherent inter-organizational interaction and mutual support aroundan EBP implementation. Such structures can include service systemcontracts with community-based organizations that deliver services,ormemorandumsof agreement that support the ICT structure. The lead-ership at the system level, and at the organizational level is needed to fa-cilitate formal agreements for the ICT structure. In addition, the ICTmodel presumes that the presence of a local seed team will create alocus of leadership at the practice level that reinforces initial commit-ments from higher-level administrators and accountability to serviceprovision and ongoing intervention fidelity. The model presumes thatthe role of the seed team, both to learn and to transmit a practice toother colleagues, will require the team to identify and resolve salient ad-aptation issues that arise in the local context. The existence of the seedteam, with its coaching and supervision responsibilities, is designed tocreate a focus on fidelity that is transmitted directly to practitionersworking in different organizations delivering the new practice. Theseed team is placed in a structural role to carry a voice of leadership

and fidelity focus throughout the course of implementation, ratherthan having this spearheaded by an external change agent. It is worthnoting that the ICT model seeks to create conditions for successful im-plementation of an identified practice change, not to influence how allpractice changes occur, although it is presumed that experience withthe model with improve local change efforts more broadly.

Initial results suggest that the ICTmodel steps do lead tomany of theintended structural supports or drivers for effective implementation.However, they also reveal that the ICT process may not sufficiently fos-ter some of the supports that receive direct attention in other processmodels. Our qualitative data suggest that including explicit processesfor regular communication in the ICT model is crucial to improve theability of involved stakeholders to identify and address potential con-flicts in ways that build trust and continued cooperation. Participants'experiences suggested that there are likely to be unexpected areas ofconflict and disagreement in any implementation effort and that clearcommunication and a problem solving orientation will facilitate effec-tive resolution of such issues.

The ICTmodel does appear to have resulted in a systemic focus on fi-delity in amanner that facilitates appropriate local adaptation. Formanyyears, the potential inflexibility and lack of local cultural relevance ofEBPs have been significant limiting concerns (Bernal, 2006; Bernal,Jimenez-Chafey, & Rodriguez, 2009; Bernal & Scharron-del-Rio, 2001;Castro, Barrera, & Martinez, 2004; Kumpfer, Alvarado, Smith, & Bellamy,2002; Lau, 2006;Matos, Torres, Santiago, Jurado, & Rodriguez, 2006). Ex-periences from participants using the ICT model with SafeCare suggestthat such concerns have some validity, but that they are addressablethrough planned implementation structures and processes. Questionsdid arise about the fit of SafeCare with local cultural nuances, particular-ly for Latino families. The seed team,with a liaison role between the EBPdeveloper and trained home visitors and families receiving services, didaddress issues of how to adapt SafeCare to local conditions while re-maining true to core components of the practice. The ICT model envi-sions this as a negotiated process between a locally based team andEBP developers who know that this team will assume responsibilityover time for system-wide fidelity maintenance. These kinds of negotia-tions were observed as the seed team recognized the need for SafeCareadaptation for the county andworkedwith SafeCare developers tomakeappropriate adaptations (Finno-Velasquez et al., under review).

At a broad level, our qualitative results largely support the role thatICT model processes play in generating structural supports for imple-mentation and sustainment of a system-wide EBP-driven quality im-provement effort. Several areas were identified as likely needingfurther attention in the implementation model, including a process fordetailing appropriate communication patterns early during implemen-tation and a process for creating forums in which differences amongparticipating organizations can be identified, discussed, and resolved.

In the presence of multiple implementation models a naturalquestion that arises concerns the conditions under which a particu-lar implementation model is particularly relevant. The ICT modelseems particularly well suited to circumstances in which an EBP isplanned for roll-out in a given organization or across a contiguousgeographic area in which authority for initiating and supportingthe effort falls under the domain of a small number of administrativeentities, in this case county child welfare services. As opposed to fo-cusing on collaborative information sharing among organizationsimplementing an EBP in different locations, or emphasizing the internalorganizational cultural and climate of implementing organizations, theICT model is relevant when local organizations are in a position to worktogether to develop a shared core infrastructure for implementing andsustaining expertise in a practice thatwill be scaled up across a broad geo-graphic area. The ICTmodel is somewhat unique in its emphasis on creat-ing implementation supports through formal and strategic structuringand staging of the implementationprocess rather than through the exten-sive involvement of an outside organization that organizes and pushesthe implementation process forward.

168 M. Hurlburt et al. / Children and Youth Services Review 39 (2014) 160–168

5. Conclusions

The ICT implementation process model developed out of the collab-orative experience of researchers and local agency partners. Use of itsimplementation processes has led to sustained and widespread use ofSafeCare, an evidence-based neglect prevention model, in one largecounty, and resulted in the phased transitioning of expertise frommodel developers to the local context. Although some aspects of the im-plementation model may benefit from enhancement, results suggestthat the process model generates strong structural supports for imple-mentation and creates conditions in which tensions between EBP struc-ture and local contextual needs can be resolved inways that support theexpansion andmaintenance of the EBPwhile preserving its potential forpublic health benefit.

Acknowledgments

This study was supported by the National Institute of Mental Healthgrants 2R01MH072961 and P30MH074678. The authors thank thecommunity-based organizations, case-managers, and supervisors thatmade this study possible. The authors declare no other conflicts ofinterest.

References

Aarons, G. A. (2006). Transformational and transactional leadership: Association with at-titudes toward evidence-based practice. Psychiatric Services, 57(8), 1162–1169.

Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R., et al.(2012). Dynamic adaptation process to implement an evidence-based child maltreat-ment intervention. Implementation Science, 7(32), 1–9.

Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model ofevidence-based practice implementation in child welfare. Administration and Policyin Mental Health and Mental Health Services Research, 38(1), 4–23.

Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in childwelfare: Service provider perspectives. Administration and Policy in Mental Health andMental Health Services Research, 34, 411–419.

Aarons, G. A., Sommerfeld, D. H., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The im-pact of evidence-based practice implementation and fidelity monitoring on staffturnover: Evidence for a protective effect. Journal of consulting and clinicalpsychology, 77(2), 270.

Aarons, G. A., Wells, R. S., Zagursky, K., Fettes, D. L., & Palinkas, L. A. (2009). Implementingevidence-based practice in communitymental health agencies: Amultiple stakeholderanalysis. American Journal of Public Health, 99(11), 2087–2095.

Allen, P., Brownson, R. C., Duggan, K., Stamatakis, K. A., & Erwin, P. C. (2012). The makingsof an evidence-based local health department: Identifying administrative and man-agement practices. Frontiers in Public Health Services and Systems Research, 1(2), 2.

Backer, T. E. (2000). The failure of success: Challenges of disseminating effective sub-stance abuse prevention programs. Journal of Community Psychology, 28(3), 363–373.

Bernal, G. (2006). Intervention development and cultural adaptation research with di-verse families. Family Process, 45(2), 143–151.

Bernal, G., Jimenez-Chafey, M. I., & Rodriguez, M.M.D. (2009). Cultural adaptation of treat-ments: A resource for considering culture in evidence-based practice. ProfessionalPsychology-Research and Practice, 40(4), 361–368.

Bernal, & Scharron-del-Rio (2001). Are empirically supported treatments valid for ethnicminorities? Toward an alternative approach for treatment research. Cultural Diversityand Ethnic Minority Psychology, 7, 328–342.

Bertram, R. M. (2008). Establishing a basis for multi-system collaboration: Systemic teamdevelopment. Journal of Sociology and Social Welfare, 35(4), 9–27.

Bond, G., Drake, R., McHugo, G., Rapp, C., & Whitley, R. (2009). Strategies for improvingfidelity in the national evidence-based practices project. Research on Social WorkPractice, 19(5), 569.

Burke, M. J. (1993). Computerized psychological testing: Impacts on measuring predictorconstructs and future job behavior. In N. Schmitt, & W. C. Borman (Eds.), Personnelselection in organizations (pp. 203–239). : Jossey-Bass.

Castro, F., Barrera, M., & Martinez, C. (2004). The cultural adaptation of prevention inter-ventions: Resolving tensions between fidelity and fit. Prevention Science, 5(1), 41–45.

Chaffin, M., Hecht, D., Bard, D., Silovsky, J. F., & Beasley, W. H. (2012). A statewide trial ofthe SafeCare home-based services model with parents in Child Protective Services.Pediatrics, 129(3), 509–515.

Chamberlain, P., Price, J., Reid, J., & Landsverk, J. (2008). Cascading implementation of afoster and kinship parent intervention. Child Welfare, 87(5), 24–48.

Crea, T. M., Crampton, D. S., Abramson-Madden, A., & Usher, C. L. (2008). Variability in theimplementation of TeamDecisionmaking (TDM): Scope and compliance with the fam-ily to family practice model. Children and Youth Services Review, 30(11), 1221–1232.

Dai, Y. S., Xie, M., Poh, K. L., & Liu, G. Q. (2003). A study of service reliability and availabilityfor distributed systems. Reliability Engineering & System Safety, 79(1), 103–112.

Damschroder, L., Aron, D., Keith, R., Kirsh, S., Alexander, J., & Lowery, J. (2009). Fosteringimplementation of health services research findings into practice: A consolidatedframework for advancing implementation science. Implementation Science, 4, 50.

Edmondson, A.C. (2004). Learning from failure in health care: Frequent opportunities,pervasive barriers. Quality and Safety in Health Care, 13(Suppl. 2), ii3–ii9.

Edmondson, A. C., & Roloff, K. S. (2009). Overcoming barriers to collaboration: Psycholog-ical safety and learning in diverse teams. Team effectiveness in complex organizations:Cross-disciplinary perspectives and approaches, 34.

Emerson, R., Fretz, R., & Shaw, L. (1995).Writing ethnographic fieldnotes. Chicago: Universityof Chicago Press.

Feldstein, A., & Glasgow, R. (2008). A practical, robust implementation and sustainabilitymodel (PRISM). Joint Commission Journal on Quality and Patient Safety, 34(4),228–243.

Ferlie, E. B., & Shortell, S. M. (2001). Improving the quality of health care in the UnitedKingdom and the United States: A framework for change. Milbank Quarterly, 79(2),281–315.

Finno-Velasquez, M. F., Fettes, D., Aarons, G. A., & Hurlburt, M. S. (under review). Culturaladaptation of an evidence-based home visitation program: Latino clients' experiences ofservice delivery during implementation. (under review).

Fixsen, D., Blase, K., Naoom, S., & Wallace, F. (2009). Core implementation components.Research on Social Work Practice, 19(5), 531.

Fixsen, D. L., Naoon, S. F., Blase, K. A., Friedman, R. M., &Wallace, F. (2005). Implementationresearch: A synthesis of the literature. Tampa: University of South Florida, Louis de laParte FloridaMental Health Institute, theNational Implementation Research Network.

Frambach, R. T., & Schillewaert, N. (2002). Organizational innovation adoption: Amulti-levelframework of determinants and opportunities for future research. Journal of BusinessResearch, Special Issue: Marketing Theory in the Next Millennium, 55(2), 163–176.

Glisson, C., & Schoenwald, S. (2005). The ARC organizational and community interventionstrategy for implementing evidence-based children's mental health treatments.Mental Health Services Research, 7(4), 243–259.

Gray, B. (1990). Building interorganizational alliances: Planned change in a global envi-ronment. Research in organizational change and development, 4, 101–140.

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of in-novations in service organizations: Systematic review and recommendations.Milbank Quarterly, 82(4), 581–629.

Grimshaw, J., Shirran, L., Thomas, R., Mowatt, G., Fraser, C., Bero, L., et al. (2001). Changingprovider behavior: An overview of systematic reviews of interventions. Medical Care,39(8), II2–II45.

Grol, R., & Grimshaw, J. (1999). Evidence-based implementation of evidence-based med-icine. Journal on Quality Improvement, 25, 503–513.

Hollenbeck, J. R., Ilgen, D. R., Sego, D. J., Hedlund, J., Major, D. A., & Phillips, J. (1995).Mutlilevel theory of team decision making: Decision performance in teams incorpo-rating distributed expertise. Journal of Applied Psychology, 80(2), 292.

Klein, K. J., Conn, A.B., & Sorra, J. S. (2001). Implementing computerized technology: Anorganizational analysis. Journal of Applied Psychology, 86(5), 811–824.

Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. Academy ofManagement Review, 21(4), 1055–1080.

Kumpfer, K. L., Alvarado, R., Smith, P., & Bellamy, N. (2002). Cultural sensitivity andadaptation in family-based prevention interventions. Prevention Science, 3(3),241–246.

Lau, A. S. (2006). Making the case for selective and directed cultural adaptations ofevidence-based treatments: Examples from parent training. Clinical Psychology: Scienceand Practice, 13(4), 295–310.

Lutzker, J. R., Bigelow, K. M., Doctor, R. M., Gershater, R. M., & Greene, B. F. (1998). Anecobehavioral model for the prevention and treatment of child abuse and neglect. In J.Lutzker (Ed.), Handbook of child abuse research and treatment (pp. 239–266). Springer.

Matos, M., Torres, R., Santiago, R., Jurado, M., & Rodriguez, I. (2006). Adaptation of parent–child interaction therapy for Puerto Rican families: A preliminary study. FamilyProcess, 45, 205–222.

Mendel, P., Meredith, L., Schoenbaum, M., Sherbourne, C., & Wells, K. (2008). Interven-tions in organizational and community context: A framework for building evidenceon dissemination and implementation in health services research. Administrationand Policy in Mental Health and Mental Health Services Research, 35(1–2), 21–37.

Nadler, D. A., & Tushman, M. L. (1977). A diagnostic model for organizational behavior. InR. J. Hackman, E. E. Lawler, L. W. N. Porter, & P S (Eds.), Perspectives on behavior inorganizations. New York: McGraw-Hill.

Palinkas, L. A., & Aarons, G. A. (2009). A view from the top: Executive and manage-ment challenges in a statewide implementation of an evidence-based practiceto reduce child neglect. International Journal of Child Health and Human Development,2(1), 47–55.

Porras, J., & Robertson, P. (1992). Organizational development. InM. Dunnette, & L. Hough(Eds.), Handbook of industrial and organizational psychology (pp. 719–822). London:Consulting Psychologists Press.

Sandelowski, M., & Barroso, J. (2003). Classifying the findings in qualitative studies.Qualitative Health Research, 13(7), 905–923.

Sosna, T., & Marsenich, L. (2006). Community development team model: Supporting themodel adherent implementation of programs and practices. Sacramento, CA: CaliforniaInstitute of Mental Health.

Stetler, C. B., McQueen, L., Demakis, J., & Mittman, B.S. (2008). An organizational frame-work and strategic implementation for system-level change to enhance research-based practice: QUERI Series. Implementation Science, 3, 30.

Trist, E. (1985). Intervention strategies for interorganizational domains. In R. Tannenbaum,N. Margulies, & F. Massarik (Eds.), Human systems development. San Francisco:Jossey-Bass.