Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Evaluating Communities of Practice and KnowledgeNetworks: A Systematic Scoping Review of EvaluationFrameworks
Kaileah A. McKellar,1,2 Kristen B. Pitzul,1 Juliana Y. Yi,1 and Donald C. Cole2,3
1Institute of Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto, 155 College Street, Suite 425, Toronto, ON M5T
3M6, Canada2Community of Practice in Ecosystems Approaches to Health, 155 College St, Toronto, ON, Canada3Dalla Lana School of Public Health, University of Toronto, 155 College St, Toronto, ON, Canada
Abstract: Communities of Practice (CoPs) are increasingly considered a part of ecohealth and other sectors
such as health care, education, and business. However, there is little agreement on approaches to evaluate the
influence and effectiveness of CoPs. The purpose of this review was to understand what frameworks and
methods have been proposed or used to evaluate CoPs and/or knowledge networks. The review searched
electronic databases in interdisciplinary, health, education, and business fields, and further collected references
and forward citations from relevant articles. Nineteen articles with 16 frameworks were included in the
synthesis. The purposes of the evaluation frameworks varied; while some focused on assessing the performance
of CoPs, several frameworks sought to learn about CoPs and their critical success factors. Nine of the
frameworks had been applied or tested in some way, most frequently to guide a case study. With limited
applications of the frameworks, strong claims about generalizability could not be made. The review results can
inform the development of tailored frameworks. However, there is a need for more detailed and targeted CoP
evaluation frameworks, as many imperative CoP evaluation needs would be unmet by the available frame-
works.
Keywords: Communities of practice, Knowledge networks, Evaluation, Scoping review
INTRODUCTION
Ecosystem approaches to health (ecohealth) is an emerging
field producing ongoing debates about environmental and
health issues, and about the development of the field itself
(Webb et al. 2010; Charron 2012; Parkes 2011). In its early
stages, the ecohealth field recognized the value of sup-
porting communities of practice (CoPs) through net-
working and capacity building activities (e.g., meetings on
specific issues such as malaria in urban areas of Africa or
regional training workshops) (De Plaen and Kilelu 2004).
CoPs are groups of people who interact on an ongoing
basis to share knowledge and expertise about common
practices, problems, or topics (Wenger et al. 2002). A
knowledge network (KN) refers to a set of connections
Electronic supplementary material: The online version of this article (doi:10.1007/
s10393-014-0958-3) contains supplementary material, which is available to autho-
rized users.
Published online: July 15, 2014
Correspondence to: Kaileah A. McKellar, e-mail: [email protected]
EcoHealth 11, 383–399, 2014DOI: 10.1007/s10393-014-0958-3
Review
� 2014 International Association for Ecology and Health
among people who provide resources to solve problems,
share knowledge, and make further connections (Wenger
et al. 2011). CoPs and KNs may occur naturally, but they
are increasingly being employed deliberately as knowledge
(KM) management tools (Wenger 2004; Cox 2005; Li et al.
2009a). Foundational to ecohealth is that ‘‘health and well-
being are the result of complex and dynamic interactions
between determinants, and between people, social and
economic conditions, and ecosystems’’ (Charron 2012).
Structures that promote alliances among multiple disci-
plines and bring in new ideas can help people understand
the interrelationships between health, ecology, and society,
and apply these understandings in practice.
The CoP model can provide opportunities to break
down professional and organizational barriers (Ranmu-
thugala et al. 2011b) and to support the learning of new-
comers to the field (Lave and Wenger 1991). It can offer a
way to translate and share tacit knowledge or ‘know how,’
which can be a valuable resource for capacity building and
the implementation of evidence-based practices (Wenger
1998; Barwick et al. 2009). CoPs offer new ways of struc-
turing collaboration in response to the challenges of com-
plex systems. With the current emphasis on partnership,
collaboration, and networks, and with the momentum of
the CoP model, new concepts, methodologies, and tech-
niques to understand the mechanisms and the potential
value of these networks are necessary. However, despite the
growing interest in, and implementation of, CoPs, little
agreement exists on approaches to evaluate their influence
(Li et al. 2009b) and limited evaluation of CoPs and
knowledge networks has occured (Li et al. 2009b; Bertone
et al. 2013). As a result, empirical evidence supporting the
effectiveness of CoPs remains limited, and even fewer
analyses investigate the mechanisms that determine effec-
tiveness (Li et al. 2009b).
Communities of practice in ecosystem approaches to
health (CoPEHs) are an application of CoPs within the
field of ecohealth. CoPEHs grew from a desire for
researchers and practitioners to share knowledge and
experience. They were developed with the central purposes
of improving research on specific ecohealth themes and
teaching others how to conduct ecohealth research (Parkes
et al. 2012). Five CoPEHs have emerged around the globe
over the last decade in Latin America and the Caribbean
(CoPEH-LAC), the Middle East and North Africa (CoPEH-
MENA), West Africa (CoPEH-AOC), South and South East
Asia (CoPEH-SSEA), and Canada (CoPEH-Canada) (see
listing and additional information on CoPEHs in Supple-
mental Material). Other partnerships are also emerging
around ecohealth and One Health. Recent additions of
regionalized ecohealth efforts include initiatives in Africa
and Oceania focused on regional conferences with potential
to catalyze ‘regional chapters’ of the International Associ-
ation for Ecology & Health. CoPs and other forms of col-
laborative networks are gaining momentum in the
ecohealth field (Parkes et al. 2012) as they provide oppor-
tunities to define and advance the field of ecohealth
through knowledge creation, enhanced learning, identity
building, and professional development. Although there is
potential for CoP and knowledge network structures to
promote learning and knowledge mobilization, limited
research exists to understand and evaluate how they work.
This review drew from Wenger and colleagues (2002,
2011) in defining CoPs and KNs; however, CoP and KN
terminology is applied differently throughout the literature.
Some authors suggest that the difference between networks
and CoPs is merely a matter of terminology (Creech et al.
2012), and definitions of KNs often align closely to those of
CoPs (Stein et al. 2001; Serrat 2010). However, despite
performing many of the same functions, other authors
distinguish basic differences, such as identity (McDermott
1999; Verburg and Andriessen 2006) or visibility (Botkin
1999). The definitions of CoP and KN used in this review
align with CoPEHs’ focus on capacity building and learn-
ing. Wenger et al.’s (2011) definition of knowledge net-
works highlights linkages that allow information flow,
knowledge creation, and learning leveraged by an individ-
ual. The original goals of CoPEHs—‘‘to foster a decen-
tralized communication mechanism to deliver key services
to its membership’’ (Flynn-Dapaah, 2003)—align with the
concept of knowledge networks. Hence, this review in-
cluded both concepts/terms, thus avoiding exclusion of
relevant evaluation frameworks and capturing an appro-
priate breadth of evaluation frameworks that would be
applicable to a variety of collaborative structures in dif-
ferent fields, including ecohealth CoPs and networks.
Several ecohealth CoPs and networks are currently
engaged in evaluation; CoPEH-LAC and CoPEH-Can are
involved in self-evaluation of their networks, informed by
social network analysis (SNA) and longitudinal qualitative
analysis (Parkes et al. 2012); Spiegel et al. (2011) developed
a framework to guide CoP activities and assess their out-
comes and impacts. Despite efforts to evaluate and
understand CoPs in Ecohealth, many questions remain,
particularly around health and social outcomes of CoPs
(Spiegel et al. 2011). Parkes et al. (2012) highlight evalua-
384 Kaileah A. McKellar et al.
tion summative questions, such as, ‘‘Would we do differ-
ently or not as well in the absence of collaboration?’’
Additionally, addressing developmental questions, such as
‘What factors contribute to successful knowledge ex-
change?’ will promote more effective practice in commu-
nities and networks. These questions can be challenging to
answer as CoPEHs exhibit elements of complex interven-
tions and have been described as dynamic and complex
learning systems (Barragan-Ocana et al. 2012). Evaluation
frameworks can help in answering these, and other, eval-
uation questions. They can guide evaluations by outlining
the approach or design to be applied, and by providing the
scheme or analytical tool for the process of assessment.
Ecohealth community members are calling for more sys-
tematic monitoring of ecohealth research and evaluation of
collaborative structures (Charron 2012; Parkes et al. 2012).
Hence, the purpose of this systematic scoping review
was to understand what frameworks and methods have
been proposed or used to evaluate CoPs and/or knowledge
networks. A scoping review approach (Arksey and O’Mal-
ley 2005; Davis et al. 2009; Levac et al. 2010) was deter-
mined to be appropriate for this review because it allowed
for a summary of the breadth and depth of the evaluation
of CoP and KNs. This review aims to provide practical
information for evaluators and network practitioners
regarding the current state of literature and to increase their
awareness of the frameworks available for adaptation to
their own ecohealth CoP and network evaluations.
METHODS
Overall Approach
As the underlying concept of CoP and/or knowledge net-
work is not well operationalized, the search terms and
databases were chosen to be inclusive of CoPs in multiple
fields and reflective of ecohealth CoPs and networks. Al-
though overlap with other types of networks, such as epi-
stemic communities (Haas 1992), practice-based networks,
and service delivery networks may occur, this review does
not include these other types because they neither fit with
the capacity building nature of CoPs nor the nature and
membership of CoPEHs and ecohealth networks. This re-
view sought to find frameworks from multiple disciplines
to facilitate learning across disciplines. For example,
frameworks for CoPs and/or knowledge networks from
fields outside of ecohealth remain relevant to ecohealth.
Hence, this review included both networks that occur
within the bounds of an organization and those that link
individuals across organizations.
Search Strategy
To identify existing evaluation frameworks for CoPs and
KNs, the review used two strategies to find studies and
reports: (1) searching eight electronic databases from
multiple disciplines and (2) snowball sampling of refer-
ences to, and citations of, relevant articles. The database
search took place in January 2013 and the eight databases—
Scopus, Web of Science, Medline, EMBASE, CINAHL,
ERIC, EconLit, and Business Review—captured frame-
works from a variety of disciplines. The search terms were
based on a preliminary scoping review and on other CoP-
related systematic reviews (Li et al. 2009b; Ranmuthugala
et al. 2011b). The terms included ‘communit* of practice’,
‘knowledge network*’ ‘network* of practice’, ‘situated
learning’, and ‘communit* of interest,’ as well as ‘evalua-
tion framework’, ‘program evaluation’, ‘evaluation meth-
ods’, and ‘logic model’. To capture the complex
intervention nature of CoPs, the review included ‘‘theor* of
change’’ and ‘realist* evaluation*’. In addition to searching
popular databases, the review examined the references and
forward citations of relevant articles. Citing articles were
found using Google Scholar and Scopus, or Web of Science,
if not found in Scopus. The ‘snowball sampling’ occurred
in October 2013.
Two reviewers (KM, KP) independently screened a
sample (4%) of article titles to pilot the screening criteria;
then one reviewer applied the criteria to the entire set
(KM). The title screening stage excluded articles with
clearly unrelated topics (e.g., artificial intelligence, clinical
trials), captured initially due to the broad nature of the
search terms. Two reviewers (KM, KP) then screened three
random samples of 100 articles (totalling 8%) to ensure the
screening criteria were clear and that they were consistently
applied (full inclusion and exclusion criteria are set out in
Supplemental Material). One reviewer (KM) then reviewed
the remaining titles and abstracts. When it was not possible
to exclude based on title and abstract, two of three
reviewers (KM, KP, JY) assessed each full text indepen-
dently. Discrepancies were discussed through to agreement;
if agreement was not achieved during initial discussion, a
third reviewer joined the process. Once the reviewers had
established relevant articles from the database search, one
reviewer (KM) screened titles and abstracts from their
references and citations, to find additional frameworks that
Evaluating Communities of Practice and Knowledge Networks 385
met the inclusion criteria. Each of the selected articles was
full text screened by two reviewers (KM, KP, JY) for rele-
vance.
English language peer-reviewed articles, conference
proceedings, and reports published after 1991 were in-
cluded. Existing reviews related to communities of practice
(Li et al. 2009b; Ranmuthugala et al. 2011b) or evaluation
frameworks (Van Eerd et al. 2011) informed the inclusion
and exclusion criteria. Articles were included based on
relevance—whether the article presented a framework de-
signed to evaluate a CoP or knowledge network. Articles
did not need to use the terms ‘CoP’, or ‘knowledge net-
work’ or ‘evaluation framework’ to describe the model and/
or methods that they presented. The screening process did
not exclude frameworks based on study quality, as the goal
was not to assess quality, but to describe frameworks that
have been proposed.
Data Extraction
Three reviewers (KM, KP, JY) iteratively tested data
extraction templates with three articles. Based on the
consistency of the results and discussions by the reviewers,
modifications for clarity were made, and additional fields
were added after each of the first two articles. One reviewer
then applied the template to the remaining articles, and two
reviewers checked 25% of the completed extractions for
accuracy and consistency. The extraction template included
multiple spreadsheets collecting data about the articles, the
types of networks, specific network information (if appli-
cable), the evaluation frameworks, and the application or
testing of the frameworks (if applicable).
RESULTS
The database search yielded 5348 original articles (Fig. 1).
After the screening process, nine articles were selected from
this set. A further ten articles were located through refer-
ences and snowball searching. The 19 articles included in
the review had 16 unique frameworks that were variously
described as, for example, ‘systematic framework for ana-
lyzing…’, ‘guidelines for assessment’, ‘diagnosis frame-
work’, or ‘conceptual framework’.
The frameworks’ publication dates ranged from 2002
to 2013, and the publications originated from a variety of
countries, including Australia, Canada, Italy, Korea, and
the Netherlands. The majority of frameworks (13 out of 16)
used CoP terminology to describe their units of analysis,
while others used the terms ‘‘knowledge network,’’
‘‘working group,’’ or a combination of these terms. In those
that focused on CoPs, the work of Wenger et al. (2002) was
commonly used or referenced to define CoPs. Most
frameworks (11 out of 16) focused on evaluating CoPs that
were set within organizations; these frameworks came from
management literature, but were not limited to a specific
sub-sector (e.g., manufacturing technology). Other frame-
works came from the fields of health (n = 3), education
(n = 1), international development (n = 1), and interdis-
ciplinary research (n = 1). The majority of the articles
provided limited descriptions of the member populations.
The review assumed that CoPs/KNs members within an
organization were employees; their disciplinary back-
grounds included finance, software development, and
engineering. Researchers, health care practitioners, and
transnational actors in health policy were included as
members of CoPs/KNs that crossed organizational
boundaries.
Purpose and Main Components of the Frameworks
Table 1 outlines the multiple aims and approaches among
the frameworks. Generally, the frameworks served one or
several of the following purposes: to assess CoPs/KNs, to
understand CoPs/KNs, and/or to promote the value of
CoPs/KNs. The majority of the frameworks were focused
on understanding CoPs/KNs rather than on assessing them
against benchmarks or specific goals. Some frameworks
addressed multiple aims; for example, to understand the
goals, strategies, or mechanism of CoPs and to measure
performance. The structures and dimensions of the
frameworks also varied. The majority used categories of
measures as bases for their models, although two studies
also included a framework that outlined the evaluation
process (Ranmuthugala et al. 2011a; Kim et al. 2012). The
right column of Table 1 outlines the main components
present in each framework.
The frameworks varied in extensiveness. Some were
quite limited in their scope while others attempt to trace
pathways between the dimensions that lead to CoP success,
and to provide a series of indicators and suggestions for
data collection and analysis (Meessen and Bertone 2012). In
some cases, authors provided limited methodological
information, making it unclear how one might operation-
alize the framework (Loyarte and Rivera 2007). Other
frameworks were quite specific and offered tools that could
386 Kaileah A. McKellar et al.
be applied in the study of CoPs and/or knowledge networks
(Verburg and Andriessen 2006; Wenger et al. 2011). Several
of the articles were analysis oriented and provided detailed
information about analysis methods and associated for-
mulae (Chu and Khosla 2009; Lee et al. 2010; Chu et al.
2012).
Dimensions Included in the Frameworks
Table 2 outlines common measurements or considerations
that are part of the frameworks. This overview provides a
sense of the types of evaluation that the frameworks pro-
pose (e.g., process or summative). This analysis explored
whether the frameworks take into consideration the inter-
nal characteristics, the external context, and the process and
outcomes of the group.
This analysis examined whether each framework con-
siders the network’s stage of development (e.g., coalescing,
stewardship) or changes over time and whether it considers
the specific goals or scope of the network or whether
incorporates measures of structure. Three frameworks
considered the stage of the CoP, each with a different ap-
proach. The framework developed by Lee et al. (2010)
provides a method for determining the stage of a CoP (i.e.,
building, growth, adaptive, close), while Zhang et al. (2011)
reported on one network’s progress through various stages.
Finally, Creech et al. (2012) drew attention to the life cycle
of the network and the importance of measuring change
over time. Only four of the frameworks incorporated the
goals or scope of the CoP/KN by including a network-level
exploration of goals/scope (Loyarte and Rivera 2007; Cre-
ech et al. 2012) or measuring progress against network
goals (Lee et al. 2010; Kim et al. 2012). Approximately half
of the frameworks included some measure of the network
structure. Earlier frameworks used descriptive measures of
structure, while frameworks that are more recent suggested
incorporating SNA into the evaluation.
Frameworks were considered to have met this criterion
if they incorporated context as defined by the frameworks’
authors. Interpretations of context varied; for example, the
Verburg and Andriessen (2006) model includes measures of
the basic structural elements of the network (e.g., size of
network, existence of subgroups) as well as organizational
support, while Scarso and colleagues (2008, 2009) consid-
ered context as the host organization’s knowledge strategy
and business environment. Context was most commonly
considered the external environment of the CoP/KN, al-
though several articles noted that the community itself is an
important context when considering the identity of mem-
bers (Verburg and Andriessen 2006; Loyarte and Rivera
2007).
The way in which context was factored into the
framework also varied. The most comprehensive inclusion
of context comes from Grootveld and Helms (2008), who
presented three categories of contextual factors (people,
organization, and system), and their fit in the organiza-
tional culture. In Conklin and Stolee’s (2008; Stolee and
Conklin, n.d.) framework, context is a knowledge exchange
dimension, and they suggested that context should be
measured at each network dimension (i.e., for the network,
network component, and implementation site). In the
frameworks proposed by Verburg and Andriessen (2006),
Scarso and colleagues (2008, 2009), and Barragan-Ocana
Figure 1. Literature search strategy: article identification and
screening for relevance
Evaluating Communities of Practice and Knowledge Networks 387
Tab
le1.
Co
Pev
alu
atio
nfr
amew
ork
sin
chro
no
logi
cal
ord
er
Au
tho
rs(y
ear)
Net
wo
rkty
pe
Sect
or/
sett
ing
Aim
of
fram
ewo
rkD
escr
ipti
on
/mai
nco
mp
on
ents
McD
erm
ott
(200
2)C
oP
Man
agem
ent/
wit
hin
org
aniz
atio
ns
To
mak
eco
nn
ecti
on
sb
etw
een
acti
viti
esan
dre
-
sult
sin
ari
goro
us,
use
ful,
and
con
vin
cin
gw
ay
Fra
mew
ork
incl
ud
es(1
)ac
tivi
ties
;(2
)
ou
tpu
ts(p
erso
nal
kno
wle
dge
,
stre
ngt
ho
fre
lati
on
ship
,an
dac
cess
to
info
rmat
ion
);(3
)va
lue
(in
div
idu
al,
team
/pro
ject
,o
rgan
izat
ion
al);
(4)
bu
sin
ess
resu
lts
Ver
bu
rgan
dA
nd
ries
sen
(200
6)
Co
P
Man
agem
ent/
wit
hin
org
aniz
atio
ns
To
get
syst
emat
icin
sigh
tsin
toth
ech
arac
teri
stic
s
and
per
form
ance
so
fC
oP
san
do
fth
eo
pin
ion
s
of
Co
Pm
emb
ers,
and
top
rovi
de
feed
bac
ko
n
the
per
form
ance
of
Co
Ps
atth
ein
div
idu
al,
gro
up
,an
do
rgan
izat
ion
alle
vel
Fra
mew
ork
stru
ctu
res
item
sin
to(1
)
char
acte
rist
ics
(fo
rms,
goal
s,ro
les)
,
(2)
pro
cess
es(a
ctiv
itie
s,co
mm
un
i-
cati
on
s,co
ord
inat
ion
and
ICT
use
),
and
(3)
ou
tco
mes
(in
div
idu
alre
-
war
ds,
gro
up
vita
lity
and
org
aniz
a-
tio
nal
pro
du
cts)
Lo
yart
ean
dR
iver
a(2
007)
Co
P
Man
agem
ent/
wit
hin
org
aniz
atio
ns
To
anal
yze
the
coh
eren
ceb
etw
een
org
aniz
atio
nal
ob
ject
ives
and
the
cult
ivat
ing
pro
cess
of
Co
Ps
(McD
erm
ott
’s(1
999)
thre
ed
imen
sio
ns)
and
to
anal
yze
the
coh
eren
ceb
etw
een
the
inte
grat
ion
of
Co
Ps
ino
rgan
izat
ion
san
dth
eir
resu
lts
(Wen
ger’
s(1
998)
bas
icco
mp
on
ents
of
Co
Ps)
Fra
mew
ork
incl
ud
es:
(1)
ob
ject
ives
(org
aniz
atio
nal
,in
div
idu
al),
(2)
pro
cess
of
cult
ivat
ion
(in
cl.
the
kin
d
of
kno
wle
dge
the
com
mu
nit
y
shar
es);
the
deg
ree
of
con
nec
tio
nan
d
iden
tity
amo
ng
mem
ber
s,an
dh
ow
clo
sely
inte
grat
edsh
arin
gkn
ow
led
ge
isw
ith
peo
ple
’sev
eryd
ayw
ork
:(3
)
inte
grat
ion
char
acte
rist
ics:
and
(4)
resu
lts
(po
siti
ve,
neg
ativ
e)
Scar
soan
dB
oli
san
i
(200
8),
Scar
soet
al.
(200
9)
Co
P
Man
agem
ent/
wit
hin
org
aniz
atio
ns
To
iden
tify
and
inte
grat
eth
em
ain
dim
ensi
on
s
shap
ing
the
crea
tio
nan
dm
anag
emen
to
fC
oP
s
Fra
mew
ork
incl
ud
esfo
ur
inte
rnal
pil
-
lars
/mai
nd
imen
sio
ns
that
gro
un
da
Co
P:
(1)
org
aniz
atio
nal
,(2
)co
gni-
tive
,(3
)ec
on
om
ic,
(4)
tech
no
logi
cal.
Itin
clu
des
two
exte
rnal
envi
ron
-
men
tal
elem
ents
:b
usi
nes
sco
nte
xt
and
kno
wle
dge
stra
tegy
388 Kaileah A. McKellar et al.
Tab
le1.
con
tin
ued
Au
tho
rs(y
ear)
Net
wo
rkty
pe
Sect
or/
sett
ing
Aim
of
fram
ewo
rkD
escr
ipti
on
/mai
nco
mp
on
ents
Co
nkl
inan
dSt
ole
e(2
008)
,
Sto
lee
and
Co
nkl
in(n
.d.)
Kn
ow
led
gen
etw
ork
Hea
lth
care
/mu
ltip
leo
rgan
izat
ion
s
To
asse
sskn
ow
led
geex
chan
geac
tivi
tyin
rela
tio
n
toth
ed
imen
sio
ns
of
evid
ence
,co
nte
xt,
faci
li-
tati
on
,an
dre
sult
s,o
nm
ult
iple
scal
eso
fn
et-
wo
rkfu
nct
ion
ing
Fra
mew
ork
isre
pre
sen
ted
by
atw
o-
dim
ensi
on
alru
bri
cw
ith
net
wo
rk
leve
ls(n
etw
ork
wid
e,n
etw
ork
com
-
po
nen
t,im
ple
men
tati
on
site
)an
d
kno
wle
dge
exch
ange
(evi
den
ce,
con
-
text
,fa
cili
tati
on
,re
sult
s)
Gro
otv
eld
and
Hel
ms
(200
8)C
oP
Ind
ust
ry/w
ith
ino
rgan
izat
ion
s
To
iden
tify
con
text
fact
ors
that
infl
uen
ceC
oP
succ
ess,
and
top
rop
ose
ap
reli
min
ary
asse
ss-
men
tto
ol
tod
iagn
ose
Co
Pco
nte
xtfa
cto
rs
wit
hin
ano
rgan
izat
ion
Fra
mew
ork
iden
tifi
es11
con
text
fact
ors
wit
hin
thre
egr
ou
ps:
(1)
peo
ple
(mem
ber
valu
e,le
ader
,tr
ust
,id
en-
tity
)(2
)o
rgan
izat
ion
(man
agem
ent
sup
po
rt,
lau
nch
stra
tegy
,K
Mst
rat-
egy,
rew
ard
syst
em),
and
(3)
syst
em
thro
ugh
wh
ich
kno
wle
dge
shar
ing
occ
urs
(med
iach
oic
e,u
sab
ilit
y).
Fi-
nal
ly,
thes
eo
ccu
rw
ith
inth
eo
rgan
i-
zati
on
’scu
ltu
re,
pre
sen
ted
asan
add
itio
nal
fact
or
Ch
uan
dK
ho
sla,
(200
9),
Ch
uet
al.
(201
2)
Co
P
Ind
ust
ry/w
ith
ino
rgan
izat
ion
s
To
anal
yze
vari
ou
sin
dex
pri
ori
ties
and
stra
tegy
pre
fere
nce
so
fC
oP
sto
un
der
stan
dth
eir
KM
cult
ure
,al
loca
tere
sou
rces
tow
ard
sth
ech
ose
n
bu
sin
ess
stra
tegy
and
mea
sure
KM
per
for-
man
cein
the
con
text
of
the
cho
sen
bu
sin
ess
stra
tegy
Fra
mew
ork
incl
ud
esfo
ur
dim
ensi
on
s:
(1)
locu
so
fle
ader
ship
,(2
)in
cen
tive
mec
han
ism
,(3
)m
emb
erin
tera
ctio
n,
(4)
com
ple
men
tary
asse
t.E
ach
has
fou
rcr
iter
ia(e
.g.,
ho
mo
gen
eity
of
mem
ber
s)w
ith
fou
ras
soci
ated
bu
si-
nes
sst
rate
gies
Lee
etal
.(2
010)
Co
P
Man
agem
ent/
wit
hin
org
aniz
atio
ns
To
asse
ssth
eco
nd
itio
no
fth
eC
oP
ind
irec
tly
and
toes
tab
lish
stag
eo
fm
atu
rity
and
aro
adm
ap
for
mo
vin
gb
etw
een
stag
es
Fra
mew
ork
iden
tifi
esa
six-
step
pro
cess
that
iden
tifi
esth
est
age
atw
hic
hth
e
Co
Pis
and
the
asso
ciat
edcr
itic
al
succ
ess
fact
ors
.In
clu
des
fou
rm
atu
-
rity
leve
ls(b
uil
din
g,gr
ow
th,
adap
-
tive
,an
dcl
ose
),11
crit
ical
succ
ess
fact
ors
,an
d28
corr
esp
on
din
gC
oP
acti
viti
es
Evaluating Communities of Practice and Knowledge Networks 389
Tab
le1.
con
tin
ued
Au
tho
rs(y
ear)
Net
wo
rkty
pe
Sect
or/
sett
ing
Aim
of
fram
ewo
rkD
escr
ipti
on
/mai
nco
mp
on
ents
Ran
mu
thu
gala
etal
.(2
011a
,b
)C
oP
Hea
lth
care
/mu
ltip
lese
ttin
gs
To
iden
tify
con
text
-mec
han
ism
-ou
tco
me
con
fig-
ura
tio
ns
that
wil
lex
pla
inth
ero
leo
fC
oP
sin
imp
rovi
ng
hea
lth
care
pra
ctic
e
Stu
dy
pro
toco
lfo
ra
real
ist
eval
uat
ion
ou
tlin
ein
fou
rst
ages
(1)
theo
ry—
CM
Oco
nfi
gura
tio
ns
gen
erat
ion
,(2
)
hyp
oth
eses
gen
erat
ion
/ref
ram
ing,
(3)
ob
serv
atio
n—
hyp
oth
eses
test
ing,
w/
surv
ey,
(4)
pro
gram
spec
ifica
tio
n—
revi
ewth
eori
esin
ligh
to
ffi
nd
ings
Wen
ger
etal
.(2
011)
Co
P
Ed
uca
tio
n/w
ith
ino
rgan
izat
ion
s
To
pro
vid
eth
efo
un
dat
ion
for
anev
alu
atio
n
pro
cess
that
can
inte
grat
eh
eter
oge
neo
us
sou
rces
and
typ
eso
fd
ata
tocr
eate
aco
mp
elli
ng
pic
ture
of
ho
wC
oP
san
dn
etw
ork
scr
eate
valu
e
for
thei
rm
emb
ers,
for
ho
stin
go
rgan
izat
ion
s,
and
for
spo
nso
rs.
Fra
mew
ork
incl
ud
esb
oth
ase
to
fre
le-
van
tin
dic
ato
rsfo
rd
ata
coll
ecti
on
and
ap
roce
ssfo
rin
tegr
atin
gth
ese
ind
icat
ors
into
am
ean
ingf
ul
acco
un
t
of
valu
ecr
eati
on
.It
incl
ud
es5
cycl
es:
(1)
imm
edia
teva
lue—
acti
viti
esan
d
inte
ract
ion
s,(2
)p
ote
nti
alva
lue—
kno
wle
dge
cap
ital
,(3
)ap
pli
edva
-
lue—
chan
ges
inp
ract
ice,
(4)
real
ized
valu
e—p
erfo
rman
ceim
pro
vem
ent,
and
(5)
refr
amin
gva
lue—
red
efin
ing
succ
ess
Zh
ang
etal
.(2
011)
Wo
rkin
ggr
ou
p
Inte
rdis
cip
lin
ary
rese
arch
/mu
lti-
na-
tio
nal
To
dem
on
stra
tep
rogr
ess
of
net
wo
rks
and
exam
ine
chal
len
ges
and
mec
han
ism
sto
add
ress
chal
len
ges
Fra
mew
ork
incl
ud
esst
ruct
ure
(net
wo
rk
size
,d
ensi
ty,
cen
tral
izat
ion
)an
d
pro
cess
des
crip
tio
nb
ased
on
stag
es
of
dev
elo
pm
ent
(in
itia
tin
g,fo
rmu
la-
tio
nag
reem
ent,
adva
nci
ng
rese
arch
,
sust
ain
ing
rese
arch
gro
up
)
Bar
raga
n-O
can
aet
al.
(201
2)C
oP
Ind
ust
ry/w
ith
ino
rgan
izat
ion
s
To
exp
lore
stru
ctu
ral
mec
han
ism
sin
use
wit
hth
e
Co
Ps,
par
ticu
larl
yth
ose
that
guar
ante
eo
rga-
niz
atio
nal
viab
ilit
yth
rou
ghth
efu
nct
ion
they
carr
yo
ut
Use
sa
viab
lesy
stem
mo
del
focu
sed
on
org
aniz
atio
ns
inte
ract
ing
wit
hth
eir
envi
ron
men
t.In
tera
ctio
ns
incl
ud
ean
op
erat
ive
elem
ent,
wh
ich
carr
ies
ou
t
the
org
aniz
atio
n’s
fun
dam
enta
l
op
erat
ion
s,an
da
met
a-sy
stem
,
wh
ich
guar
ante
esth
eo
rgan
izat
ion
’s
fun
ctio
ns
asa
‘wh
ole
’
390 Kaileah A. McKellar et al.
Tab
le1.
con
tin
ued
Au
tho
rs(y
ear)
Net
wo
rkty
pe
Sect
or/
sett
ing
Aim
of
fram
ewo
rkD
escr
ipti
on
/mai
nco
mp
on
ents
Cre
ech
etal
.(2
012)
Net
wo
rks
and
/or
Co
Ps
Inte
rnat
ion
ald
evel
op
men
t
To
stre
ngt
hen
per
form
ance
asse
ssm
ent
and
imp
rove
men
to
fn
etw
ork
s,an
dto
iden
tify
star
tin
gp
oin
tsin
layi
ng
ou
ta
mo
red
etai
led
rese
arch
agen
da
for
futu
rew
ork
Fra
mew
ork
incl
ud
esfo
ur
po
ints
(1)
focu
san
dex
ten
sive
nes
so
fth
en
et-
wo
rk,
(2)
evo
luti
on
of
the
stru
ctu
re
of
the
net
wo
rko
ver
tim
e(3
)so
cial
cap
ital
wit
hin
the
net
wo
rk,
and
(4)
acti
viti
esan
do
utc
om
eso
fth
en
et-
wo
rk,a
nd
the
shar
edva
lue
crea
ted
by
the
net
wo
rk
Kim
etal
.(2
012)
Co
P
Man
agem
ent/
wit
hin
org
aniz
atio
ns
To
iden
tify
curr
ent
kno
wle
dge
-sh
arin
gac
tivi
ty
stat
us
ina
Co
P
Fra
mew
ork
ou
tlin
esse
vera
lco
nse
cuti
ve
stag
eso
f(1
)le
arn
ing
abo
ut
the
net
-
wo
rk(S
NA
):cl
assi
fyin
gm
emb
ers
(bal
ance
dp
laye
r,eg
ois
tic
pro
pag
ato
r,
ego
isti
cre
ceiv
eran
dkn
ow
led
geis
o-
lato
r),
(2)
clas
sify
ing
the
net
wo
rk
(act
ive
com
mu
nit
y,sp
read
ing
com
-
mu
nit
y,le
arn
ing
com
mu
nit
y,in
ac-
tive
com
mu
nit
y),
and
then
(3)
alig
nin
git
wit
ha
set
of
Co
Pst
rate
gies
(e.g
.,in
crea
seaw
aren
ess
of
mem
ber
s’
kno
wle
dge
)
Mee
ssen
and
Ber
ton
e(2
012)
Co
P
Hea
lth
po
licy
/tra
nsn
atio
nal
To
anal
yze
the
role
that
kno
wle
dge
(wh
ich
is
pro
du
ced
,cr
eate
dan
dm
anag
edth
rou
ghth
e
Co
Ps)
pla
ysin
the
pro
cess
of
sele
ctin
gan
d
imp
lem
enti
ng
hea
lth
po
lici
es,
wh
ich
intu
rn,
may
hav
eim
pac
tso
nh
ealt
ho
utc
om
esva
lued
by
citi
zen
s
Fra
mew
ork
retr
aces
the
cau
sali
typ
ath
bet
wee
na
seri
eso
fd
imen
sio
ns
that
mem
ber
sac
tiva
teto
ensu
reth
eC
oP
fun
ctio
ns
wel
l.T
he
six
dim
ensi
on
s
(ava
ilab
lere
sou
rces
,st
rate
gies
to
mo
bil
ize
reso
urc
es,
kno
wle
dge
man
-
agem
ent
pro
cess
es,
exp
ansi
on
of
kno
wle
dge
,kn
ow
led
ge-b
ased
po
lici
es
and
pra
ctic
es,
bet
ter
hea
lth
and
wel
-
fare
ou
tco
mes
)co
nta
ins
elem
ents
and
/or
sub
-ele
men
tsan
dan
asso
ci-
ated
set
of
ind
icat
ors
Evaluating Communities of Practice and Knowledge Networks 391
et al. (2012), context/environment is one of three or four
major components of their model. Loyarte and Rivera
(2007), Chu and colleagues (2009, 2012), and Wenger et al.
(2011) gave mention to the importance of context, but do
not state specific measures.
Understanding the processes and activities of a net-
work is a key step in understanding the factors that make it
effective and the ways through which it achieves outcomes.
Frameworks that propose documentation of a CoP/KNs
activities or their process of development were included.
Most frameworks measured some dimension of network
activity. Several of these frameworks (Scarso and Bolisani
2008; Ranmuthugala et al. 2011a; Meessen and Bertone
2012) focused on the mechanisms that create effective
networks. Eight of the frameworks included measures of
CoP/KN outcomes. Of the frameworks that included out-
comes, most focused only on the positive effects of net-
works; two frameworks (Loyarte and Rivera 2007; Scarso
and Bolisani 2008) included both positive outcomes and
negative outcomes or costs of the networks. The level of
impact refers to whether outcomes are measured by mul-
tiple scales; seven of the frameworks do. Four focused on
individual levels and group and/or organizational levels. Set
in the healthcare environment, Conklin and Stolee (2008)
and Stolee and Conklin (n.d.) included outcomes at the
implementation site, at the network-component (CoP) le-
vel and at the wider network level. Wenger et al. (2011) had
a different interpretation of level of outcome; they include
levels of immediate value, potential value, applied value,
and realized value. Creech et al. (2012) suggested that
assessment took into consideration the level of value cre-
ation outlined by Wenger et al. (2011).
Application/Testing of Evaluation Frameworks
Nine of the studies applied or tested their frameworks in
some form (Table 3). Seven of these frameworks were fo-
cused on CoPs within organizations; two were focused on
groups that crossed organizational boundaries. The level of
reporting with respect to the application or testing process
was varied. What was learned about CoPs/KNs were more
commonly reported than what was learned about the
framework or the evaluation process. Only two of the
articles reported any modifications to the frameworks after
they were piloted (Verburg and Andriessen 2006; Grootveld
and Helms 2008). The majority of the articles framed the
testing or application as a case study. Three of the studies
tested or pilot tested their frameworks; this was possibleTab
le1.
con
tin
ued
Au
tho
rs(y
ear)
Net
wo
rkty
pe
Sect
or/
sett
ing
Aim
of
fram
ewo
rkD
escr
ipti
on
/mai
nco
mp
on
ents
Yap
and
Ro
bb
en(2
012)
Co
P
Ind
ust
ry/w
ith
ino
rgan
izat
ion
s
To
asse
ssth
ead
ded
valu
eo
fn
etw
ork
edle
arn
ing
thro
ugh
on
lin
eco
mm
un
itie
san
dn
etw
ork
sfo
r
bu
sin
esse
s
Th
em
od
elli
nks
retu
rn-o
n-i
nve
stm
ent
for
net
wo
rks
of
pra
ctic
ete
chn
olo
gy
toth
ere
turn
-on
-in
vest
men
tfo
rth
e
reso
urc
esan
dti
me.
Spec
ific
com
po
-
nen
tsar
eo
nli
ne
dia
log
(so
cial
-cap
i-
tal,
lear
nin
g);
nar
rati
vecy
cles
of
mea
sure
men
t(i
mm
edia
te,
po
ten
tial
,
app
lied
,re
aliz
ed,
and
refr
amin
g);
and
mea
sure
so
fle
arn
ing
(sh
are-
ho
lder
valu
e,cu
sto
mer
enga
gem
ent,
and
emp
loye
eex
per
ien
ce)
392 Kaileah A. McKellar et al.
because each framework was, or incorporated, an analytical
tool. Two of the case study articles provided limited
information about how the framework was applied, making
it difficult to understand how the framework and evalua-
tion process could be applied or transferred to another
setting (Loyarte and Rivera 2007; Scarso et al. 2009).
Many of the frameworks provided general guides and
approaches to evaluate CoPs/KNs; therefore, determining
their test validities was not relevant. Four of the frame-
works provided quantitative analytical tools; only Verburg
and Andriessen (2006) reported on the process (i.e., per-
forming scale analysis and reliability checks) and the results
of determining the tools’ validity. One study made claims
about the reliability of the framework without demon-
strating how this was determined (Chu et al. 2012). The
majority of studies suggested that their frameworks would
benefit from additional empirical investigation to validate
and refine them.
Articles also discussed the generalizability of the
frameworks, making suggestions about where else and how
the framework could be used; however, due to lack of
testing, there was limited evidence of their applicability.
Four articles suggested that the frameworks were applicable
outside of the contexts in which they were tested. For
example, Kim et al. (2012) suggested that, although their
framework focused on CoP activities, the lessons learned
would be applicable to any knowledge-sharing activities
within an organization. Five of the articles concluded that
further application of the frameworks would be needed to
understand their applicability to other settings. Some of the
frameworks’ authors highlighted the need to test within
different types of communities or settings.
All articles concluded that the frameworks were useful
in assessing or understanding dimensions of CoP/KNs.
While many authors acknowledged the limitations of their
studies in demonstrating validity, several studies high-
Table 2. Overview of CoP evaluation framework components
Authors (year) Goals/scope Stage/time Context Structure Process/activities Outcomes Level of impact
McDermott (2002) 4 4
Verburg and Andriessen (2006) 4 4 4 4
Loyarte and Rivera (2007) 4 4 4 4 (+/-)
Scarso and Bolisani (2008),
Scarso et al. (2009)
4 4 4 4 (+/-)
Conklin and Stolee (2008), Stolee and
Conklin (n.d.)
4 4 4 4
Grootveld and Helms (2008) 4
Chu and Khosla (2009),
Chu et al. (2012)
4 4
Lee et al. (2010) 4 4
Ranmuthugala et al. (2011a) 4 4 (SNA)
Wenger et al. (2011) 4 4 (SNA) 4 4 4
Zhang et al. (2011) 4 4 (SNA) 4
Barragan-Ocana et al. (2012) 4 4 4
Creech et al. (2012) 4 4 4 (SNA) 4 4 4
Kim et al. (2012) 4 4(SNA)
Meessen and Bertone (2012) 4 4 4
Yap and Robben (2012) 4 4 4
Stage/time: considered the stage of development of the network (e.g., coalescing, stewardship), or changes in the network over time. Goals/scope: considered
the specific goals or scope of the network. Context: incorporated context in the framework (context as defined by authors). Structure: incorporated measures
of structure in the framework (e.g., size of network). SNA: used social network analysis to measure the structural dimension of the network. Process/activities:
included measures of network activities or the processes of development. Outcomes: incorporated measures of outcomes in the framework. -/+: considered
both positive and negative outcomes and/or costs of the network. Level of impact: incorporated outcome measures and different scales (e.g., individual,
network, organization)
Evaluating Communities of Practice and Knowledge Networks 393
Tab
le3.
Eva
luat
ion
fram
ewo
rkap
pli
cati
on
/tes
tin
gap
pro
ach
es
Au
tho
rs(y
ear)
Ho
wfr
amew
ork
was
app
lied
or
test
ed
Mem
ber
s’d
isci
pli
nar
yb
ackg
rou
nd
Met
ho
ds
Th
efr
amew
ork
s’au
tho
rs‘
con
clu
sio
nfr
om
app
lica
tio
n
Ver
bu
rgan
dA
nd
ries
sen
(200
6)
Pil
ot
test
edto
ol
on
7C
oP
sw
ith
ina
larg
eo
rgan
izat
ion
Soft
war
ed
evel
op
men
t
Dat
aco
llec
tio
nco
nsi
sted
of
thre
ep
arts
:(1
)
mem
ber
s-o
nli
ne
qu
esti
on
nai
reo
fm
emb
ers)
;
(2)
coo
rdin
ato
r-in
terv
iew
wit
hC
oP
coo
rdin
a-
tor;
(3)
con
text
-ch
eckl
ist
for
hig
hle
vel
key
info
rman
tsfr
om
org
aniz
atio
n
Refi
nem
ents
mad
eto
item
sin
the
too
l.T
he
asse
ssm
ent
too
lp
rovi
ded
au
sefu
l
met
ho
dfo
rq
uan
tita
tive
lyan
alyz
ing
Co
P
effe
ctiv
enes
s
Lo
yart
ean
dR
iver
a(2
007)
Use
dfr
amew
ork
tost
ud
yth
eex
per
i-
ence
so
f15
org
aniz
atio
ns
cult
ivat
ing
Co
Ps
Mu
ltip
led
isci
pli
nes
(e.g
.,b
iosc
ien
ce,
edu
cati
on
,fi
nan
ce)
Do
cum
ent
revi
ewan
do
bse
rvat
ion
wer
eu
sed
to
coll
ect
info
rmat
ion
.A
uth
ors
con
tras
ted
theo
ry
of
Co
Ps
wit
hex
per
ien
ces,
and
anal
yzed
chal
-
len
ges
inb
uil
din
gC
oP
s
Th
eau
tho
rsd
rew
no
con
clu
sio
ns
abo
ut
the
fram
ewo
rkit
self
.B
ased
on
the
re-
sult
so
fth
est
ud
y,th
eycr
eate
da
cult
i-
vati
on
mo
del
togu
ide
thin
kin
gw
ith
in
org
aniz
atio
ns
Scar
soet
al.
(200
9)C
ase
stu
dy
of
Co
Ps
wit
hin
ala
rge
mu
ltin
atio
nal
org
aniz
atio
n
Exp
lora
tio
nan
dp
rod
uct
ion
ino
il
com
pan
y
Qu
alit
ativ
ean
dex
plo
rato
ryca
sest
ud
yw
ith
dir
ect
invo
lvem
ent
of
staf
f
Th
ere
sult
sw
ere
enco
ura
gin
gfo
rth
e
fram
ewo
rks
use
fuln
ess
for
aso
un
dan
d
syst
emat
icin
vest
igat
ion
of
the
fun
c-
tio
nin
gan
dcr
itic
alsu
cces
sfa
cto
rso
f
exis
tin
gC
oP
s.H
ow
ever
,fu
rth
erin
ves-
tiga
tio
nis
nee
ded
tore
fin
ean
dva
lid
ate
the
fram
ewo
rk
Co
nkl
inan
dSt
ole
e(2
008)
Cas
est
ud
yo
fa
kno
wle
dge
exch
ange
acti
vity
that
incl
ud
eda
Co
P
Hea
lth
care
(sen
iors
’h
ealt
h)
Co
llec
tio
nm
eth
od
sin
clu
de
do
cum
ent
revi
ew,
tele
ph
on
ein
terv
iew
wit
hkn
ow
led
geb
roke
r,
and
emai
lsu
rvey
of
acti
vity
par
tici
pan
ts
Th
eau
tho
rsco
ncl
ud
edu
sin
gel
emen
tso
f
the
PA
RiH
Sfr
amew
ork
(Kit
son
etal
.
1998
)an
dco
nsi
der
ing
pro
cess
and
im-
pac
tsin
term
so
fle
vel
of
op
erat
ion
tob
e
valu
able
.A
dd
itio
nal
test
ing
and
revi
ew
wo
uld
ben
efit
the
eval
uat
ion
mo
del
Gro
otv
eld
and
Hel
ms
(200
8)
Cas
est
ud
yo
fC
oP
sw
ith
ina
larg
e
mu
ltin
atio
nal
org
aniz
atio
n
En
gin
eeri
ng
Sem
i-st
ruct
ure
din
terv
iew
wit
hC
oP
mem
ber
s,
each
fact
or
fro
mfr
amew
ork
info
rmed
an
inte
rvie
wq
ues
tio
n
Th
efr
amew
ork
cou
ldb
eu
sed
toas
sess
the
con
text
ual
fact
ors
that
con
trib
ute
to
Co
Psu
cces
sw
ith
inan
org
aniz
atio
n.
Fra
mew
ork
was
use
ful
inp
ract
ice,
bu
ta
sin
gle
case
was
no
tsu
ffici
ent
tova
lid
ate
394 Kaileah A. McKellar et al.
Tab
le3.
con
tin
ued
Au
tho
rs(y
ear)
Ho
wfr
amew
ork
was
app
lied
or
test
ed
Mem
ber
s’d
isci
pli
nar
yb
ackg
rou
nd
Met
ho
ds
Th
efr
amew
ork
s’au
tho
rs‘
con
clu
sio
nfr
om
app
lica
tio
n
Ch
uan
dK
ho
sla
(200
9),
Ch
uet
al.
2012
Sin
gle
(200
9)an
dco
mp
arat
ive
(201
2)
case
stu
dy
of
Co
Ps
wit
hin
larg
e
org
aniz
atio
ns
Tec
hn
olo
gyre
sear
ch
Qu
esti
on
nai
red
istr
ibu
ted
tosp
ecifi
cu
nit
sw
ith
in
two
org
aniz
atio
ns
Th
ere
sult
sge
ner
ated
wer
eco
nsi
sten
tw
ith
goal
san
dst
rate
gies
of
bo
thth
esu
rvey
ed
org
aniz
atio
ns.
Th
ism
od
elw
asth
eref
ore
‘‘re
liab
le,
pra
ctic
al,
and
suit
able
for
ado
pti
on
by
mu
ltin
atio
nal
kno
wle
dge
-
bas
edo
rgan
izat
ion
sin
gen
eral
.’’
Lee
etal
.(2
010)
Pil
ote
dm
od
elw
ith
thre
eC
oP
sw
ith
in
on
eco
mp
any
Fin
ance
Co
Pm
emb
ersu
rvey
toid
enti
fycu
rren
tst
atu
s
amo
ng
pre
-defi
ned
mat
uri
tyst
ages
and
an
anal
ytic
alh
iera
rch
yp
roce
ssto
det
erm
ine
rela
-
tive
imp
ort
ance
of
crit
ical
succ
ess
fact
ors
Fra
mew
ork
is‘‘
ho
list
ic,
syst
emat
ic,
com
-
pre
hen
sive
wit
hm
atu
rity
stag
esfo
r
nav
igat
ing
Co
Ps’
’h
ow
ever
veri
fica
tio
n
thro
ugh
add
itio
nal
app
lica
tio
nis
nee
ded
Zh
ang
etal
.(2
011)
Sin
gle
case
stu
dy
wit
hSN
A(e
valu
atio
n
bu
ilt
aro
un
dsp
ecifi
cn
eed
so
fn
et-
wo
rk)
Mu
ltip
led
isci
pli
nes
rela
ted
tod
igit
al
gove
rnm
ent
rese
arch
Self
-ad
min
iste
red
soci
alm
atri
xq
ues
tio
nn
aire
,
and
syst
emat
ico
bse
rvat
ion
san
dre
flec
tio
ns
of
the
exp
erie
nce
mad
eb
yth
ete
amm
emb
ers
Th
est
ud
yw
asab
leto
iden
tify
chal
len
ges
and
key
enab
lin
gfa
cto
rso
fth
eev
alu
ated
gro
up
;h
ow
ever
,th
ege
ner
aliz
abil
ity
wil
l
req
uir
ead
dit
ion
alca
sest
ud
ies
fro
m
vari
ou
sse
ttin
gs
Kim
etal
.(2
012)
Ap
pli
edm
eth
od
olo
gyto
59C
oP
sin
a
com
pan
yw
ell
kno
wn
for
kno
wle
dge
man
agem
ent
Man
ufa
ctu
rin
g
Tra
cked
on
lin
elo
go
fC
oP
acti
vity
(po
sts
to
web
site
),u
sed
SNA
Th
efr
amew
ork
cou
ldb
eap
pli
edin
oth
er
org
aniz
atio
ns
and
the
less
on
sle
arn
ed
fro
mre
sult
sco
uld
be
use
ful
toan
alyz
e
kno
wle
dge
-sh
arin
go
rkn
ow
led
getr
ans-
fer
acti
viti
esin
any
area
of
ano
rgan
i-
zati
on
Evaluating Communities of Practice and Knowledge Networks 395
lighted the utility of the frameworks when drawing con-
clusions about their value. Authors provided examples of
how their case studies were able to inform strategic action
in the CoPs (Grootveld and Helms 2008) or shared the
results that came from using their frameworks; for example,
after implementing framework-based recommendations,
CoPs became more productive in their contributions to
online forums (Kim et al. 2012).
DISCUSSION
The body of literature pertaining to CoPs is growing;
however, relatively few studies examine the evaluation of
these structures. This study demonstrates the variety of
purposes, scopes, and methodological approaches taken in
evaluating CoPs and/or KNs. A key observation is that,
despite finding approximately half of the frameworks
through references and citations, there have been limited
adaptations and adoptions from previous studies. Fur-
thermore, reports on applications of the frameworks have
been restricted to the framework developers.
Addressing Common Challenges in Evaluation
CoPs can, and CoPEHs do, exhibit elements of a complex
intervention, bringing about certain evaluation challenges.
A complex intervention has characteristics of nonlinearity,
emergence, adaptation, uncertainty, dynamic interactions,
and co-evolution (Patton 2011). Evaluators often catego-
rize interventions as simple, complicated, complex, or
chaotic (Rogers 2008; Patton 2011). There are common
challenges in evaluating CoPEHs and other complex
interventions: the emergent nature of activities and out-
comes, multiple scales of impact, long timelines of impact,
hard-to-measure/intangible outcomes, and challenges of
attribution/contribution. The evaluation frameworks re-
viewed here dealt with the challenges in different ways, with
some authors incorporating flexibility and suggesting
exploratory approaches.
CoPs are dynamic structures, which poses challenges
for evaluation. Networks change, and with new members
taking part, the foci of the CoPs can shift, not only in terms
of the practices, but also in terms of forms of engagement
and the structures of the networks themself. Similarly,
CoPEHs have changing membership and shifting priorities.
Some of the frameworks reviewed are likely too rigid to fit
the dynamic nature of the CoPEHs. More adaptable
frameworks could support evaluations of emergent net-
works (e.g., Conklin and Stolee 2008; Scarso and Bolisani
2008; Creech et al. 2012; Meessen and Bertone 2012).
Theory-based approaches and realist evaluations (Ranmu-
thugala et al. 2011a) would also be appropriate. It is nearly
impossible to isolate the effects of a CoP, and to determine
causality for a specific program. Designs solely focused on
attribution of CoP effects or employing counterfactuals are
not suitable for evaluating CoPs. More plausible are pre-
post designs, such as that suggested by McDermott (2002),
to understand the value of CoPs within organizations.
CoPs can foster conditions for innovation (Lesser and
Storck 2001; Verburg and Andriessen 2006; Chu and
Khosla 2009). Innovation, however, is difficult to measure
given its tendency to occur in quantum leaps (Perrin 2002).
Trust, social capital, communication of tacit knowledge,
and learning are other examples of important CoP out-
comes that are challenging to measure. While some
frameworks do not address the intangible or hard-to-
measure aspects of CoPs, Wenger et al. (2011) suggested
using member narratives. Another challenge for evaluation
is the multiple scales on which CoPs operate and can expect
to see outcomes. Several of the frameworks did incorporate
measures at multiple levels of impact, for example,
exploring value at the individual member level, the network
level, and the organizational level. Nevertheless, the Co-
PEHs work at additional scales and multiple timelines of
impact. Some outcomes will occur in the short-term, such
as connections developed among researchers, whereas
ecological, health, and/or social impacts are likely to hap-
pen in the longer term.
Application to Ecohealth CoPs and Networks
Multiple forms of information from the frameworks can
provide guidance for the evaluation of Ecohealth CoPs and
Networks. Several frameworks proposed detailed sets of
indicators (Wenger et al. 2011; Meessen and Bertone 2012),
which might be helpful for ecohealth researchers and
practitioners evaluating their CoPs, particularly for
understanding value creation and influence on policy and
practice. Verburg and Andriessen (2006) and Wenger et al.
(2011) provide tools that with some adaption could guide
data collection. Other frameworks are less detailed, but
highlight key considerations that would be useful in eval-
uating CoPs; for example, Creech suggests the importance
of CoP goals and stage of development. In addition to the
guidance for CoP and KN assessments provided by the
396 Kaileah A. McKellar et al.
frameworks, evaluators can learn from the results of the
evaluations. Several of the articles and reports included
potential mechanisms of how CoPs and/or KNs work and
some suggested critical success factors that could inform
the development of a CoP. The level of evidence supporting
the mechanisms and success factors varies, in some cases,
information was based on case studies, in others a list was
generated from a literature review. One article did not
specify the source or methods used in determining success
factors. Examples of success factors include financial re-
source, developing trust, opportunities for face-to-face
meeting, and self-selected membership (as cited by Zhang
et al. 2011; Ranmuthugala et al. 2011a). Keeping in mind
that mechanism can produce different results depending on
the context, a realist informed review would be useful in
understanding which are applicable to CoPs and networks
working within the field of ecohealth.
It is worth highlighting several frameworks that would
be most relevant to Ecohealth CoPs and networks. As
mentioned previously, Verburg and Andriessen (2006)
provide a tool and member questionnaire that could be
useful in collecting relevant information with respect to
CoP activities and value. Given that knowledge-to-action is
one of the principles of Ecohealth, the Conklin and Stolee
(2008) framework includes relevant structure for evalua-
tion questions. Furthermore, the multiple levels of analysis
could accommodate the nodal structure and heterogeneous
membership of the CoPEHs. Exploring impacts at the
implementation site acknowledges that knowledge flow
within a network can have impacts beyond the members or
their organization. Attention to context is stressed by the
Grootveld and Helms (2008) framework, though not all
contextual factors would be relevant to CoPs and KNs
working within the field of ecohealth due to the within
organization context. While the framework by Chu and
colleagues (2009, 2012) is situated in a business context, the
focus on strategies for CoP improvement could inform an
approach to evaluate Ecohealth CoPs and networks. The
Wenger et al. (2011) framework would be particularly
useful in understanding the value created for members and
could be adapted to include value-creation outside of the
CoP/KN, thereby promoting a greater understanding of the
social, health, and environmental impacts. Meessen and
Bertone’s (2012) knowledge-focused framework brings in
potentially relevant long-term goals of influencing knowl-
edge-based policy decisions and practices and better health
outcomes.
While the reviewed frameworks provide useful starting
places, the specific needs of ecohealth evaluations cannot be
met by simply applying one of the frameworks. For
example, the members of the CoPEHs hold a number of
roles; these include researcher, trainee/student, ecohealth
practitioner, and funder. Many of the frameworks in this
review did not include approaches for heterogeneous
membership. This is not surprising as homogeneity has
been used as a characteristic to describe CoPs (Fischer
2001), and many of the frameworks came from within
organizations with less heterogeneity. Notably, transdis-
ciplinarity was not broached in the reviewed frameworks.
One would therefore need to look elsewhere to answer
questions regarding the measurement of transdisciplinarity
outcomes, such as the cross-fertilization of ideas.
Acknowledging the diversity of CoPs and KNs implies
acknowledging a required equal diversity in approaches to
their evaluation. While some of the frameworks reviewed
here provided enough detail (including data collection
tools), it would be inappropriate to apply any tool without
considering community type (e.g., as outlined by Dube
et al. (2006) in their typology of virtual CoPs), the learning
needs of the community, the key evaluation questions, and
the desired application of evaluation processes and results.
Future research should explore how evaluation frame-
works, methods, and approaches can complement partic-
ular types of CoPs and their objectives, thereby informing
insights on how best to evaluate a particular CoP or KN.
An exploration that matches frameworks to CoPs could be
useful in answering questions such as, ‘Are there key out-
comes that should be measured in the early stages (e.g.,
relationship building) that are less relevant after a CoP is
well established?’ or ‘What approaches are best for dis-
persed multinational, multi-linguistic networks, such as
several of the CoPEHs?’
Strengths and Limitations
This review builds on existing literature and can inform
practices of the CoPEHs, as unique CoPs. A recent article
by Bertone et al. (2013) reports findings from a scoping
review of frameworks for assessing CoP performance that
would inform a transnational health policy CoP evaluation
model. The reviews differ in the methods used; Bertone
et al.’s search methods were not systematic or exhaustive,
they based their criteria for inclusion on relevance to a
research question that was defined post hoc. In addition to
Evaluating Communities of Practice and Knowledge Networks 397
assessment frameworks, Bertone et al. (2013) also included
literature on limitations and success factors of CoPs. While
there is some overlap with frameworks in this review, our
review examined a broader scope of evaluation frameworks
from multiple disciplines and explicitly dealt with testing of
evaluation frameworks. Two other systematic reviews of
CoP-related topics exist. Li et al. (2009b) examined the
definition and use of CoPs in the health and business
sectors and the effectiveness of CoPs in the health care
sectors; however, the authors were not able to find studies
that met their criteria for determining CoP effectiveness.
Ranmuthugala et al. (2011b) looked at how and why CoPs
are established in the health care sector; however, this re-
view did not focus on evaluation, but did conclude with a
call for use of evaluation to assess the role of CoPs in health
care.
The strengths of this review are in the systematic nat-
ure of the search, data extraction, and assessment of the
utility of the results as a starting point for future evalua-
tions of CoPs in ecosystem health. This review is limited, as
not all potentially useful frameworks or measures could be
included. Lessons could be gained from frameworks that
focus on networks of organizations rather than individuals
(Creech and Ramji 2004) or that focus on other forms of
collaborative partnerships, such as research networks
(Fenton et al. 2007). Furthermore, books are also potential
sources of guidance in evaluating CoPs and KNs; for
example, Stein et al. (2001) highlight important evaluation
questions for knowledge networks to consider.
Conclusion
This scoping review synthesized current evidence that can
inform the evaluation practices of any CoPs or knowledge
networks. The systematic scoping review methodology al-
lowed for a breadth of framework exploration. The review
provides a base of conceptual frameworks, data collection
tools, indicators, analytical approaches, and key consider-
ations. Although application of the frameworks to the
COPEHs will vary, one can glean insights from each. With
the current emphasis on networked knowledge exchange
and capacity building, it is important to understand the
potential costs and benefits of CoPEHs as learning net-
works, and the ways in which benefits can be maximized.
By understanding the value of CoPs and the ways in which
they work, ecohealth researchers and practitioners will be
able to improve knowledge exchange and mutual learning,
towards the shared goal of societal benefit.
REFERENCES
Arksey H, O’Malley L (2005) Scoping studies: towards a meth-odological framework. International journal of social researchmethodology 8:19–32
Barragan-Ocana A, Quijano-Solıs A, Vega-Dıaz G, Sanchez-Lara B(2012) Communities of practice: A focus from complex systems.International Journal of General Systems 41:741–755
Barwick MA, Peters J, Boydell K (2009) Getting to uptake: Docommunities of practice support the implementation of evi-dence-based practice? Journal of the Canadian Academy of Childand Adolescent Psychiatry 18:16
Bertone MP, Meessen B, Clarysse G, Hercot D, Kelley A, KafandoY, Lange I, Pfaffmann J, Ridde V, Sieleunou I (2013) Assessingcommunities of practice in health policy: A conceptual frame-work as a first step towards empirical research. Health ResearchPolicy and Systems 11:39
Botkin JW (1999) Smart business: How knowledge communities canrevolutionize your company, New York: The Free Press
Charron DF (2012) Ecohealth Research in Practice: InnovativeApplications of an Ecosystem Approach to Health, Insight andInnovation in Development, New York: Springer and Ottawa,Canada: International Development Research Centre
Chu M-T, Khosla R (2009) Index evaluations and businessstrategies on communities of practice. Expert Systems withApplications 36:1549–1558
Chu M-T, Khosla R, Nishida T (2012) Communities of practicemodel driven knowledge management in multinational knowl-edge based enterprises. Journal of Intelligent Manufacturing23:1707–1720
Conklin J, Stolee P (2008) A model for evaluating knowledgeexchange in a network context. CJNR (Canadian Journal ofNursing Research) 40:116–124
Cox A (2005) What are communities of practice? A comparativereview of four seminal works Journal of Information Science31:527–540
Creech H, Ramji A (2004) Knowledge networks: Guidelines forassessment. International Institute for Sustainable Develop-ment. http://www.iisd.org/pdf/2004/networks_guidelines_for_assessment.pdf. Accessed 7 Oct, 2013.
Creech H, Laurie M, Paas L, Parry J-E (2012) Performanceimprovement and assessment of collaboration: Starting pointsfor networks and communities of practice. International Insti-tute for Sustainable Development. http://www.iisd.org/pdf/2012/performance_improvement_networks.pdf. Accessed 7 Oct2013.
Davis K, Drey N, Gould D (2009) What are scoping studies? Areview of the nursing literature International journal of nursingstudies 46:1386–1400
De Plaen R, Kilelu C (2004) From multiple voices to a commonlanguage: Ecosystem approaches to human health as anemerging paradigm. EcoHealth 1:SU8–SU15
Dube L, Bourhis A, Jacob R (2006) Towards a typology of virtualcommunities of practice. Interdisciplinary Journal of Informa-tion, Knowledge, and Management 1:69–93
Fenton E, Harvey J, Sturt J (2007) Evaluating primary care re-search networks. Health Services Management Research 20:162–173
Fischer G (2001) Communities of interest: Learning through theinteraction of multiple knowledge systems. In Proceedings of the24th IRIS Conference.
398 Kaileah A. McKellar et al.
Flynn-Dapaah K (2003) Ecosystem approaches to human healthglobal community of practice: Report on the design phase consul-tations, Ottawa: International Development Research Centre
Grootveld A, Helms R (2008) Development and application of afactor framework to diagnose possible failure of communities ofpractice. In The 9th European Conference on Knowledge Man-agement: Eckm 2008.
Haas PM (1992) Introduction: epistemic communities andinternational policy coordination. International organization46(1):1–35
Kim S, Hong J, Suh E (2012) A diagnosis framework for identi-fying the current knowledge sharing activity status in a com-munity of practice. Expert Systems with Applications 39:130893
Kitson A, Harvey G, McCormack B (1998) Enabling the imple-mentation of evidence based practice: a conceptual framework.Quality in Health care 7:149–158
Lave J (1991) Wenger E, Cambridge university press: Situatedlearning: Legitimate peripheral participation
Lee J, Suh E-h, Hong J (2010) A maturity model based copevaluation framework: A case study of strategic cops in a Koreancompany. Expert Systems with Applications 37:2670–2681
Lesser EL, Storck J (2001) Communities of practice and organi-zational performance. IBM systems journal 40:831–841
Levac D, Colquhoun H, O’Brien KK (2010) Scoping studies:advancing the methodology. Implementation Science 5:1–9
Li LC, Grimshaw JM, Nielsen C, Judd M, Coyte PC, Graham ID(2009) Evolution of Wenger’s concept of community of prac-tice. Implementation Science 4:11
Li LC, Grimshaw JM, Nielsen C, Judd M, Coyte PC, Graham ID(2009) Use of communities of practice in business and healthcare sectors: A systematic review. Implementation Science 4:16
Loyarte E, Rivera O (2007) Communities of practice: A model fortheir cultivation. Journal of Knowledge Management 11:67–77
McDermott R (1999) Nurturing three-dimensional communitiesof practice. Knowledge Management Review 11:26–29
McDermott R (2002) Measuring the impact of communities whathappens when you need to show proof that the communities areworking and are worth the investment? This article explainshow to make sense of a chain of community activities, outcomesand value in the language of the business Knowledge Manage-ment Review 5:26–29
Meessen B, Bertone MP (2012) Assessing performance of com-munities of practice in health policy.
Parkes MW (2011) Diversity, emergence, resilience: Guides for anew generation of ecohealth research and practice. EcoHealth8:137–139
Parkes MW, Charron DF, Sanchez A (2012) Better together: field-building networks at the frontiers of ecohealth research. In: Eco-health Research in Practice: Innovative Applications of an EcosystemApproach to Health, Charron DF (editor), Insight and Innovationin Development, New York, NY: Springer and Ottawa, Canada:International Development Research Centre, pp 231–253.
Patton MQ (2011) Developmental evaluation: Applying complexityconcepts to enhance innovation and use, New York: The GuilfordPress
Perrin B (2002) How to—and how not to—evaluate innovation.Evaluation 8:13–28
Ranmuthugala G, Cunningham FC, Plumb JJ, Long J, Georgiou A,Westbrook JI, Braithwaite J (2011) A realist evaluation of therole of communities of practice in changing healthcare practice.Implementation Science 6:49
Ranmuthugala G, Plumb JJ, Cunningham FC, Georgiou A, West-brook JI, Braithwaite J (2011) How and why are communities ofpractice established in the healthcare sector? A systematic reviewof the literature BMC Health Services Research 11:273
Rogers PJ (2008) Using programme theory to evaluate compli-cated and complex aspects of interventions. Evaluation 14:29–48
Scarso E, Bolisani E (2008) Communities of practice as structuresfor managing knowledge in networked corporations. Journal ofManufacturing Technology Management 19:374–390
Scarso E, Bolisani E, Salvador L (2009) A systematic frameworkfor analysing the critical success factors of communities ofpractice. Journal of Knowledge Management 13:431–447
Serrat O (2010) Knowledge solutions: tools, methods and ap-proaches to drive development forward and enhance its effects.Asian Development Bank.
Spiegel J, Breilh J, Beltran E, Parra J, Solis F, Yassi A, et al (2011)Establishing a community of practice of researchers, practitio-ners, policy-makers and communities to sustainably manageenvironmental health risks in Ecuador. BMC internationalhealth and human rights 11:S5
Stein J, Stren R, Fitzgibbon J, MacLean M (2001) Networks ofKnowledge: Collaborative Innovation in International Learning,Toronto: University of Toronto Press
Stolee P, Conklin J (n.d.) Assessment of impact of SHRTNactivities. http://seniorshealthknowledgenetwork.ca/sites/seniorshealthknowledgenetwork.ca/files/Model_to_guide_the_evaluation_of_SHRTN_2_.pdf. Accessed 7 Oct 2013.
Van Eerd D, Cole D, Keown K, Irvin E, Kramer D, BrennemanGibson J, Kohn M, Mahood Q, Slack T, Amick B III, Phipps D,Garcia J, Morassaei S (2011) Report on knowledge transfer andexchange practices: a systematic review of the quality and types ofinstruments used to assess kte implementation and impact, Tor-onto: Institute for Work & Health
Verburg RM, Andriessen J (2006) The assessment of communitiesof practice. Knowledge and Process Management 13:13–25
Webb JC, Mergler D, Parkes MW, Saint-Charles J, Spiegel J, Walt-ner-Toews D, Yassi A, Woollard RF (2010) Tools for thoughtfulaction: the role of ecosystem approaches to health in enhancingpublic health. Canadian Journal of Public Health 101:439–441
Wenger E (1998) Communities of Practice: Learning, Meaning, andIdentity, Boston: Cambridge University Press
Wenger E (2004) Knowledge management as a doughnut: shapingyour knowledge strategy through communities of practice. IveyBusiness Journal 68:1–8
Wenger E, McDermott R, Snyder WM (2002) Cultivating Com-munities of Practice: A Guide to Managing Knowledge, Boston:Harvard Business School Press
Wenger E, Trayner B, de Laat M (2011) Promoting and assessingvalue creation in communities and networks: a conceptualframework. http://wenger-trayner.com/wp-content/uploads/2011/12/11-04-Wenger_Trayner_DeLaat_Value_creation.pdf.Accessed 7 Oct 2013.
Yap R, Robben J (2012) Measuring the value of online commu-nities and networks of practice for business. In: Proceedings ofthe 8th International Conference on Networked Learning.
Zhang J, Luna-Reyes LF, Nakashima M, Gil-Garcia JR, Sayogo DS,Mellouli S (2011) Building and sustaining a transnational andinterdisciplinary research group: lessons learned from a NorthAmerican experience. In: System Sciences (HICSS), 2011 44thHawaii International Conference on System Sciences.
Evaluating Communities of Practice and Knowledge Networks 399