AComplex-SystemsOrientation

toEvaluationthatSupportsa

CultureofHealth

Welcome!

Thisconceptpaperisaboutthesimplicityontheothersideofcomplexity.Itmaynotbeenoughtogiveyourlifefor,buthopefullyit’senoughtoinspireyoutothinkanddialogueabouthowyouframeyourapproachtoevaluationandtoconsiderwhatreframingmightbehelpful.

OnewayevaluatorsmaycontributemoreusefullytocreatingaCultureofHealthisbyunderstandingfundamentaldistinctionsbetweenthehierarchicalandnetworkedstructuresofsystemsandhowtheyareintertwinedincomplexsystems.Distinguishingbetweenthesetwostructureshelpsevaluatorsgroundtheirevaluationsinthefundamentalstructuresthatareofimportanceataparticulartimeandplaceastheysupportevaluationusersintheirintentiontotransformsystemstoservethepublicgood.

Everynowandthena[person’s]mindisstretchedbyanewideaorsensation,andnevershrinksbacktoitsformerdimensions.—OliverWendellHolmes,Sr.

Iwouldnotgiveafigforthesimplicitythissideofcomplexity,butIwouldgivemylifeforthesimplicityontheothersideofcomplexity.—OliverWendellHolmes,Sr.

vThePaper’sOrientation 2

vThePaper’sPurpose3

vKeyPointsofthePaper4 vAComplex-Systems

OrientationtoEvaluation27

vEvaluationGroundedinHierarchicalStructures29

vEvaluationGroundedinNetworkedStructures30

vContrastofEvaluativeFocusinHierarchical&NetworkedStructures33

v Culture35vACultureofHealth36vACultureofHealthActionFramework37

v ExampleofanEvaluationSituationandApproach38

TableofContents

vWhatWeMeanandWhyItMatters 6

vEvaluation&EvaluativeThinking 7

vSystemic&SystematicInquiry 9

vKnowledge&KnowledgeManagement10

vSystems 11vComplexSystems13vSystemsChange 21vSystemConditions22v ImplicitBias23vWhyItMatters25

01 02 03INTRODUCTION

Pages 34–41

Pages 26–33Pages 5–25

Pages 1–4 WhatWeMeanandWhyItMatters

AComplex-SystemsOrientationtoEvaluation

AComplex-SystemsOrientationtoEvaluationthatSupportsaCultureofHealth

RESOURCES

Page 42-43

v ImplicationsforEvaluationandEvaluators43

vReferences 45

vAcknowledgements46

Pages 44–46

CalltoAction

04

RW.17.rf.SystemsWhitePaper.2-14

Introduction

v ThePaper’sOrientationv ThePaper’sPurposev KeyPointsofthePaper

|1

ThePaper’sOrientation

Sinceevaluation’sinceptionasafieldinthe1960s,thefieldhasbeendominatedintheU.S.byan

implicitbiastowardhierarchicalstructuresthattendtobestable,predictable,andcontrollable.Such

controlleddynamicsarethebasisofhierarchiesandfactorymodelsoforganization.Inaboutthe

1980s,systemstheoristsrangingfrommathematiciansandphysiciststobiologists,sociologists,

psychologists,andecologistsbegantounderstandcomplexsystemsashavingnetworkedstructures

withemergentdynamicsthatinternallychange,formingflexible,complexnon-linearpatterns

(sometimescalled“self-organizing”dynamics).Thecapacityofcomputerswasbecomingsufficientto

modeltheseemergentpatternsinareasonableamountoftime.

Withtheexpandedcapacityofcomputersandsocialnetworks,evaluatorscanmorepowerfully

contributetounderstandinghowtoinfluencesocialconditionsforthepublicgoodbygroundingtheir

evaluationpracticeinthefundamentalsofcomplexsystems.Influencingsocialconditionsthrough

influencingcomplexsocialsystemsisattheheartofcreatingaCultureofHealth.

Introd

uctio

n

|4

ThePaper’sPurposeThisconceptpaperprovidesawaytothinkaboutcomplexsystems.Thepaper

countersevaluation’sbiastowardassumingthatanevaluationisoccurring

withinhierarchicalstructureswiththeircontrolledandstablepatternsof

movement(“dynamics”).Thisdominantapproachpaysinsufficientattention

totheemergentdynamicsofnetworkedstructuresthatcharacterizemany

systemsthatalsocontributetoacultureofhealth.

Theevaluationfield’stendencytofocuslargelyonhierarchicalstructuresis

evidenced,forexample,inprogramlogicmodelsthatfocusondesired

measurableoutcomesandtheprogram’sefficiencywithintheconfinesofa

hierarchicallystructuredsystem.Suchevaluationstypicallypaylimited

attentiontootherresultsoftheinterventionthatarenotwithinthecontrolof

theprogram.

Acomplex-systemsorientationshiftsevaluators’attentiontomultiplepatterns

overtimebasedontheintertwinednatureofhierarchicalandnetworked

structureswithincomplexsystems.Webeginbylookingatcomplexsystems

andthenreturntoacultureofhealthandsuggestsomeimplicationsfor

evaluatorsandevaluationusers.

Thespecificsofhowtoapplysuchanorientationtoevaluationareinneedof

conversationanddevelopment.Thispaperisintendedtostimulate

conversationsratherthanprovidefinalanswers.

Purpose:v Toencourageevaluativeinquiryofcomplexsystems

v Togroundevaluativeinquiryofcomplexsystemsin:~ understandinghowhierarchicalandnetworkedstructurescreatecomplexsystems~ recognizingevaluation’simplicitbiastowardhierarchicalsystems

Introd

uctio

n

|3

KeyPointsofthePaper

Acultureofhealthisacomplexsystem

Evaluationtosupportthecreationofacultureofhealthneedstobecongruentwiththe

fundamentalnatureofcomplexsystems

Dominantevaluationapproacheshaveanimplicitbiascountertoacomplexsystemsorientation

TheRobertWoodsJohnson

Foundation’sCultureofHealth

involvesfourresearch-basedaction

areas.Theyinteracttocreate

improvedpopulationhealth,well-

being,andequity.

Thefouractionareasare:

• makinghealthasharedvalue

• fosteringcross-sector

collaborationtoimprovewell-

being;

• creatinghealthier,moreequitable

communities;and

• strengtheningintegrationof

healthservices,andsystems.

Asevaluators,wecanassist

evaluationusersinseeing,

understanding,andinfluencingthe

developmentofacultureofhealth

byusingorientationsand

consequentmethodologiesthatare

congruentwiththenatureofthe

entangledstructuresanddynamics

ofcomplexsystems.

Thispaperilluminateswhyand

howtoleveragetwofundamentally

differentsystemstructures—

hierarchicalandnetworked—when

usingacomplex-systems

orientationinevaluation.

Today’sdominantevaluation

methodologicalorientationhas

animplicitbiastowardsystems

withhierarchicalstructures.This

dominantorientationfailsto

recognizetheimportantand

fundamentaldifferencesbetween

thehierarchicalandnetworked

structuresthatintertwinetoform

complexsystems.

Introd

uctio

n

|2

WhatWeMeanby:v Evaluation&EvaluativeThinking

v Systematic&SystematicInquiryv Knowledge&Knowledge

Managementv Systems

v ComplexSystems

v SystemsChangev SystemConditionsv ImplicitBias

01andWhyitMatters|5

WhatWeMeanandWhyItMattersWhatWeMeanby:v Evaluation&EvaluativeThinkingv Systemic&SystematicInquiryv Knowledge&Knowledge

Managementv Systemsv ComplexSystemsv SystemsChangev SystemConditionsv ImplicitBias

Oursocietyisdeeplyrootedinthinkingintermsof“stable

systems,”bywhichmostpeoplemeanhierarchicalstructures

withcontrolleddynamics.Tothinkaboutcomplexsystems,we

needtoaddnewtermsandconceptsandredefinesome

existingterms.

Thissectionexplainshowtothinkdifferentlyaboutconcepts

thatseemfamiliarbuthaveadifferentmeaninginthecontext

ofcomplexsystems.Itisespeciallyimportanttograspthe

differencesbetweenstructuresthattendtohavecontrolled,

stablepatternsofdynamics(hierarchicalstructures)andthose

thattendtohaveflexible,emergentdynamics(networked

structures). ??…[E]veryinstanceoflanguageusemakesitsownsmallcontributiontoreproducingand/ortransformingsocietyandculture,includingpowerrelations.(Fairclough&Wodak,1998).

01.W

hatW

eMeanandWhyItM

atters

|6

01.W

hatW

eMeanandWhyItM

atters

??Evaluation&EvaluativeThinking

Evaluation issystematic

inquiry to determinemerit

(quality),worth (value),and

significance (importance)

aboutsomeentityorpurpose.

AccordingtotheEncyclopediaofEvaluation:

Evaluation isanappliedinquiryprocessforcollectingandsynthesizing

evidencethatculminatesinconclusionsaboutthestateofaffairs,value,

merit,worth,significance,orqualityofaprogram,product,person,policy,

proposal,orplan.

Conclusionsmadeinevaluationsencompassbothanempiricalaspect(that

somethingisthecase)andanormativeaspect(judgementaboutthevalue

ofsomething).Itisthevaluefeaturethatdistinguishesevaluationfrom

othertypesofinquiry,suchasbasicscienceresearch,clinicalepidemiology,

investigativejournalism,orpublicpolling.” (Mathison,2005)

Ofparticularimportanceisthelaststatementthatemphasizesthat“itisthe

valuefeaturethatdistinguishesevaluationfromothertypesofinquiry.”

Thefield ofevaluationas practicedintheU.S.andmostothercountriesis

groundedinanassumptionthathumanshaveequalrights.Alsoevaluationis

donewithinrealworldsettings,notlaboratorysettingsdisconnectedfrom

thecontextinwhichtheentityistofunction.

Thusthedefinitionofevaluationdoesnotendwithsystematicinquiry,but

usesthisinquirytodeterminemerit,worth,and/orsignificance.

|7

01.W

hatW

eMeanandWhyItM

atters

EvaluativeThinkingExample

Evaluativethinkinginvolves

balancingmerit,worth,and

significance.

Forexample,assumeyouarelookingforanewcar.

Youfindanelectriccarthatisofveryhighquality(merit).Itisofhighvalue

(worth)toyoubecauseitalignswellwithyourcommitmentto

environmentalresponsibility.However,ifyourdaughtersuddenlybecomes

seriouslyillandyouneedtospendyourmoneyonhealthcareforher,the

cardropsinitsimportance(significance)becauseyourdaughterismore

importanttoyouthanacar.

Traditionally,evaluatorshavemadeevaluativedeterminationsbasedon

theirsystematicinquiryandpredeterminedcriteria.Asthecomplexityof

societyhasbecomemoreevident,evaluatorshavebeguntoengageafull

rangeofstakeholderswithmultipleperspectivesindeterminingmerit,

worth,andsignificance.Additionally,theyarelearningtohelppeople

balancemerit,worth,andsignificanceincomplexsystems.

??

|8

01.W

hatW

eMeanandWhyItM

atters

Systemic&SystematicInquiry

• Engageinsystematicand

systematic inquiry.

• Considercontent,context,conditions, andconnectivity.

• Useinterdisciplinary knowledge.

Therearemanydifferentmethodologiesforengaginginsystematic

inquiryinwaysthatarebothethicalandofhighquality.Wearenot

addressingtherangeofmethodologiesinthispaper;ratherthe

focusisondoinganinquiryfromasystemicperspective.

Threekeypointsforaninquiryare:

• Theevaluatorengagesinsystemic aswellassystematicinquiry.

• Inthinkingsystemicallyandsystematically,theevaluatorlooks

broadlyanddeeplyintothecontent,context,conditions,and

connectivityofthesystemsinvolved.

• Thetheoryofchangeguidingasystems-orientedevaluation

typicallyneedstorelyonastronginterdisciplinarybodyof

knowledge.

??

|9

01.W

hatW

eMeanandWhyItM

atters

Knowledge&KnowledgeManagement

Knowledgemanagement (KM)isthe

processofcreating,sharing,using

andmanagingtheknowledgeand

informationofanorganization.(Girard&Girard,2015)

Complexsystemsinvolvemulti-disciplinaryknowledge.Accessing

andsynthesizingabroadrangeofknowledgetoassistevaluation

usersindevelopinganappropriatetheoryofchangebecomesan

increasinglyimportantroleforevaluatorswhenworkingwith

complexsystems.Evaluatorsandevaluationuserstypicallyare

familiarwithamorelimitedrangeofdisciplinaryknowledgethan

whatisrelevantwhenworkingincomplexsystems.

Thispaperdoesnotaddressknowledgemanagementbutitisa

criticallyimportantaspectofengaginginsystems-oriented

evaluation.Fortunately,moretechnologically-supportedmeansfor

knowledgesynthesisandmanagementareemergingbutmuchmore

developmentisneededforefficientmanagementoftheknowledge

relevanttoseeing,understanding,andinfluencingcomplexsystems.

??

|10

01.W

hatW

eMeanandWhyItM

atters

SystemsAsystemisacollectionofentitiesthatare“seen”bysomeoneasinteractingtogethertodosomething.(OpenUniversity,2011)

Severalaspectsofthisdefinitionareimportant:

• Itisacollectionofentities;multipleelementsareinvolved.

• Differentpeoplemaybeseeingadifferentcollectionofentitiesinteractingtogether,thus,seeingadifferentsystem.

• Thequotationmarksaround“seen”remindusthatthesesystemsmaybeseeninourmind’seyeratherthannecessarilybeingseenphysically.Systemsaremostoftenconceptualmodelsratherthanphysicalentities.

• Theentitiesareinteracting.Therearebothconnectionsanddynamicsinthoseinteractions.Noticemovement andenergy.

• Theinteractingentitiesaredoingsomething.Theycreatearesultthatisdifferentthantheresultsofthesumoftheparts.

• Inevaluation,desiredoutcomesaretypicallythoughtofastheresultthatasystemproduces.However,thisistoonarrowofadefinitionofwhatasystemdoes.Asystemislikelytodomorethanproducetheoutcomessomeoneishopingtosee.Forexample,theresultcanbechangesinpatternswithinthesystems,notonlyspecificproducts.

??

System: Asystemisa

collectionofentitiesthatare

“seen”bysomeoneas

interacting togethertodo

something. (OpenUniversity,UK)

Systemsthinking:Thinkingin

termsofsystemsratherthan

theparts;thinkingabout

movement,dynamics,and

patterns acrosstimeand

locations;thinkinginsystems

|11

01.W

hatW

eMeanandWhyItM

atters

SystemsThinking

System: Asystemisacollection

ofentitiesthatare“seen”by

someoneasinteracting

togethertodosomething.(OpenUniversity,UK)

Systemsthinking:Thinkingin

termsofsystemsratherthan

theparts;thinkingabout

structures,dynamics,and

patterns acrosstimeand

locations;thinkinginsystems

Regardinghumanandotherlivingsystems,systemsthinkinginvolvesthinkingaboutstructures,dynamics,andpatternsacrosstimeandlocations.Sometimesitisreferredtoas“thinkinginsystems”.

Lookatthepicturesonthispage.Whatsystemsdoyou“see”?(Thepicturesmayshowonlyapartofasystem.)Forexample,thepictureoffoodinamarketmayleadyoutothinkofasystemof:

• transportationfromfarmtomarket

• theflowofmoneyrelatedtoagriculture

• theinteractionofpeopleatalocalfarmers’market

Whatsystemsdoyouseeinthispictureandtheotherpicturesbelow?

??

|12

01.W

hatW

eMeanandWhyItM

atters

ComplexSystems

Complexsystems:Complexsystems

aremassivelyentangledsystems that

includebothhierarchicaland

networkedstructures.

Complexsocialsystemsarecomposedofmassivelyentangledstructures.Forexample,theyinvolveinterconnected families; hierarchical,bureaucraticorganizations; andnetworksofsmallformalandinformalgroups. Notonlyaretheyentangled,theyarealsoinmotionwithcomplexpatterns.

Fromatheoreticalperspective,complexsystemscanbethoughtofasanentangledmixofhierarchicalandnetworkedstructures(seefollowingpages).Hierarchicalandnetworkedstructuresareassociatedwithdifferencesintheirdynamics,i.e.,theirpatternsofmovementovertime;theirpredictability;andtheirknowability.

Complexsystemshavebeenstudiedextensivelyindifferentdisciplineswithanaccumulationofknowledgeaboutthedifferencesinhowtheyfunction.TheReferencelistincludesafewsourcesthatwehavefoundhelpfulwhenapplyingsystemsconceptstoevaluation.

Understandingacomplexsystembearssimilaritiestocomprehendingthisopticalillusion,whichcontainsimagesofbothayoungwomanandanoldwoman.Bothimagesarewithinthepicturebutifoneisoverlydominantinourview,wecan’tseetheother.

Wewanttogettothepointwherewecanmoveourfocusbackandforthbetweenhierarchicalandnetworkedstructurestoseethecomplexsystemasawhole.

??

|13

C

ThinkinginComplexSystems

Lookagainatthepicturesyoulookedatonanearlierpage.Thinkabouttheentanglementsofmultiplesystemsthat

youcanseeinthepictures.

??

01.W

hatW

eMeanandWhyItM

atters

|14

HierarchicalStructuresAmajoradvancewiththeIndustrialRevolutionwasthedevelopmentoffactoriesandassemblylines.Thesewereoftenaccompaniedbyhierarchicalorganizationalstructurestomanagepeople.Theunderlyingassumptionisthatplanningisusefulbecausethereisapredictablerelationshipbetweenactivitiesandresults.Suchstructuresaredesignedtogiveconsiderablecontroltoanorganizationalleaderormanager.Thesestructuresarefurthercontrolledbyrulesandpolicies;theytendtohavepartsthatcanbereplaced.

Whenaskedtoidentifyasystem, wehavefoundthatpeopleusuallynamesometypeofhierarchicalstructure(e.g.,theeducation“system”).Hierarchicalstructuresarepervasiveingovernment,business,andthenonprofitorganizationworlds.Theyhavemanysubsystemswithinthemthataresimilarlystructured(e.g.,themanagementsystem, theaccountingsystem).

Whenconditionsarestableandpredictableandthefunctionofthestructureistoproducewell-definedproductsoroutcomes,thistypeofstructurecanworkwell.Forexample,wewantcontrolandstabilityin anorganization’s payrolldepartmenttoassuretimelyandaccuratepaychecks.

Theassumptionsofstabilityandpredictabilityserveasthebasisforthinkingthatoneprogramcanbereplacedbyanotherwithinasocialsystemandthat,ifeachpartisworkingwell,theoverallsystemwillalsobeworkingwell.

Hierarchicalstructurestendtocreate controlleddynamicswithstable,predictablepatterns.Theyhaveconstraintswithinoraroundthemthatcreatetheirpredictability.

??

Manytoolsandmethodsexistforunderstandingandinfluencinghierarchicalstructures.Thesetoolsandmethods,basedonlinearmodelsofcauseandeffect,includestrategicplanning,settingspecificoutcomes,andusingresearchdesignssuchasrandomizedcontrolledtrials(RCT).

01.W

hatW

eMeanandWhyItM

atters

|15

01.W

hatW

eMeanandWhyItM

atters

HierarchicalStructureswithinComplexSystems

|16

v FundamentalStructure:hierarchical,mechanistic

v Dynamic:controlled,stable,predictable,“regular,”causeandeffect

v Patterns:linear,repetitive,interchangeableparts

v Examples:bureaucracies,assemblylines

01.W

hatW

eMeanandWhyItM

atters

Thecomplexityscienceshavetheirbeginningscenturiesago,buttheideasstartedtoreceiveseriousattentioninthescientificcommunityinaboutthe1980s(Capra,2014).Theunderstandingandapplicationsoftheconceptsofnetworkedstructureswithemergent(or“self-organizing”)dynamicsarestillintheirearlystagesofdevelopment.Theconceptoriginatedinthephysical,biological,andecologicalsciences;ithasmathematicalunderpinnings.

Inanetworkedstructure,weseetheemergenceofnew,unexpectedprocessesandpatternsofmovement.Thesehigher-levelemergentdynamicsunexpectedlyresultfromtheactionsofamultiplicityofsmalloccurrenceswithinthesystem.Thesmalloccurrenceswerenotplannedtocreatetheneworder.Theemergenceofthenewisnotcontrolledbyasingleentity,butresultsfromsemi-independentinteractionsofmanyagents.Theyareadaptingtoeachotherandtheenvironmentasawhole.Theseinteractionscanformpatternsovertimeandlocations.Thosepatternsmaynotbevisibletotheagentsthemselvesandthetimingofthepatternsisoftenunpredictable.

Networkedstructuresmaybedisturbedbyexternalinfluencesbuttheytendtobewithoutexternalcontrol.NetworkedstructuressuchasthehumannetworksontheInternetdisplayemergentdynamics:theyareflexibleandadaptive;noleaderisincontrol.

Networkstructuresfunctionfarfromequilibrium.Theycancreateemergentdynamics withnew,unexpected,self-organizingpatterns,properties,and/orprocesses.

??NetworkedStructures

|17

01.W

hatW

eMeanandWhyItM

atters

Networkedstructures also:

• aresensitivetoinitialconditions.Incomplexsystems,verysmalldifferencesininitialconditionscanhaveadisproportionatelylargeimpactonfutureevents.

• areinfluencedbyiterativefeedback.Theagentsare“learning”fromoneanotherandthecontextastheygetsignalsfromotheragentsandmodifytheirbehavior.

• co-evolve.Co-evolutionreferstotheinterdependentevolutionoftwoormorestructureswithinalargerecologicalsystem.Cooperation,competition,andinterdependenceinrelationtothesamelimitedresourcescreatefeedbackamongtheentitiesthroughthestructuresinacomplexsystem.

• haveactiveboundaries.Asystem’sboundariesarecreatedbythestructureitself.Dynamicstructuressuchasanetwork’shaveactiveboundariesastheyinteractwiththeirenvironments.

• mayoperatefarfromequilibrium.Theycanhaveawiderangeofemergentdynamicswithpatternsofmovementthatmaybeverycomplex.

• canshiftintonewstructures.Theshiftintoaneworganizationalstructurecanarisefromfluctuationsamplifiedbycertaintypesoffeedbackloops.

Networkedstructurescancreateemergentdynamicswithnew,unexpected,self-organizingpatterns,properties,and/orprocesses.

??CharacteristicsofNetworkedStructures

|18

01.W

hatW

eMeanandWhyItM

atters

v Fundamentalstructure:networksofnodesandconnections

v Dynamic:flexible,creative,emergent

v Pattern:co-evolving,semi-independentagentsinfluencingnearbyagents

v Examples:networks,socialmovements,movementsofmovements,swarmingofbirds,schoolsoffish

NetworkedStructureswithinComplexSystems

|19

01.W

hatW

eMeanandWhyItM

atters

Seldomwouldanidentifiedsystemhavethepurefeaturesofahierarchicalornetworkedstructureasdescribedabove.Rathersocialsystemsareamixofstructuresalthoughtheymaytendinonedirectionortheotherintheiroverallcharacter.Hierarchicalandnetworkedstructuresarelensesthroughwhichweviewcomplexsystems.Eachlensrevealsdifferentaspectsofthecomplexityandgivesusinsightintoitsfunctioning.

Complexlivingsystemsgenerallyhavesomeaspectsthatcanbemodeledwellbythinkingofthemashierarchicaland/ornetworkedstructures.

Organizationalpartnershipsandcommunitiesaregoodillustrationsofcomplexsystemswherehierarchicalandnetworkedstructuresbothcanbeidentifiedfairlyreadily.

ComplexSystems:EntangledHierarchicalandNetworkedStructures

|20

01.W

hatW

eMeanandWhyItM

atters

SystemsChangeWe oftenhearpeopletalkaboutwantingtobringaboutchangeinsystemsthroughtheirinterventions.Whenengaginginevaluationrelatedtochangingsystems,weneedtobeveryawarethatthesystemsnaturallyarealreadyinmotionandchanging.Weneedtobeawarethattheinterventionisnotenteringastaticsystem.Interventionsarelikelytochangethepattern ofchangeratherthansimplycreatechange.

Humansystemsarecontinuallychanginginandofthemselvesandthroughtheirconnectionswithothersystems.Theyarelivingsystemsthatarecontinuallychangingwhetherornotthereisanintervention.Thisisespeciallytrueofsystemswithnetworkedstructures.

Interventionssuchasprograms,policies,people,andinitiativesarechangingthepatternsofchange thatarealreadypresentinthesystems.

Itisalsoimportanttorecognizethataninterventionmaybeaffectingmoresystemsthanthosefocusedonbytheintervenor.Thecomplexityoftheinterconnectedsystems,theunpredictabilityofpatternsofchangeinthesesystems,andthesustainabilityofchangesallneedtobetakenintoaccountwhenevaluatinganinterventionincomplexlivingsystems.

Complexlivingsystemsarecontinuallychangingbasedontheirinternaldynamicsaswellasoutsideinfluences.

The influenceofaninterventiondependsontheexistingstructuresanddynamicsofthesystemsbeingaffected.

Programs,policies,peopleandotherinterventionsinfluencethepatternsofchangethatarealreadypresentinthesystems.

??

|21

01.W

hatW

eMeanandWhyItM

atters

SystemConditionsSystemstheoristshaveidentifiedcommonfeaturesorconditionsofsystems.DrawingontheworkofWilliamsandHummelbrunner(2009)andEoyang(1997),weuseInterconnections,Boundaries,andPerspectivesasabasicwaytoframetheconditionsthatdeterminethenatureofhumansystemsandtheirspecialrelevancetoevaluation.Asweconductacomplex-systemsevaluation,weneedtopayspecialattentionto:

• Understandinginterconnections(interrelationships)

• Maintaininganawarenessofboundaries

• Recognizingmultipleperspectives

Interconnections,boundaries,andperspectivesarenotisolatedfromoneanother.Asyouchangeone,youaffectanother.Thinkofthethreeconditionsastheanglesofatriangle.Asyouchangeoneangleyouaffecttheothertwo.

Theseconditionsarerelevantatanyphaseofevaluation.Theycanalsobeusedtounderstandcomplexsystemswiththeirentangledcombinationsofhierarchicalandnetworkedstructures.

Understandingtheinterplayamongtheseconditionsispartofthechallengeofthinkinginsystems.Weneedtoapproachitwiththe“forestandthetrees”mindset.Wehavetobeabletoshiftinandoutoffocusbetweenseeingtheforestandseeingtheindividualtrees.

Thesesystemconditionscanbeusedtohelpyouidentifythestructuresandtheirdynamicsinthesystemsthatyouaregoingtobeevaluatingandchooseevaluationdesignsthatarecongruentwiththesestructures.

Humansystemshavethreegeneralcommonsystemconditions:

Interconnections

Boundaries

Perspectives

I

B P

??

|22

01.W

hatW

eMeanandWhyItM

atters

ImplicitBiasAnimplicitbiasisdifferentfromasuppressedthought thatsomeonemightconceal.Implicitbiasesare“biasesthatareactivatedinvoluntarilyandbeyondourawarenessorintentionalcontrol”(Staats,Capatosto,&Contractor,2015,p.5).Implicitbiasesoftenshowupinourlanguage.

“Unwantedimplicitbiasescanbemitigated….Bytakingthetimetounderstandyourpersonalbiases,youcanbegintomitigatetheireffects....Implicitbiaseshavehugeimplications;thusitisimportanttoidentifyyourownbiasesandthenactivelyengageindebiasingtechniquestoaddressthem.” (Staats,Capatosto,&Contractor,2015,p.5).

Thereisgrowingattentiontostructuralracismastheresultofimplicitbiases.Structuralracismreferstothenormalizationandlegitimizationofinstitutionalpoliciesandpracticesthatroutinelyadvantagewhiteswhileproducingcumulativeandchronicadverseoutcomesforpeopleofcolor.Structuralracismbecomesasystemofitsownduetotheseinterconnectedorganizedsystems.

Theterm“implicitbias”canalsobeappliedtowhatishappeninginevaluationeventhoughtheimpactisverydifferentthanthatresultingfromstructuralracism.

Whenitcomestoevaluation,acriticallyimportantimplicitbiasistowardamodelofsystemswithhierarchicalstructuresandcontrolleddynamics.Thispervasivebiashasmanyimplicationsforevaluation.

“Implicitbiasisconcernedwithunconsciouscognitionthatinfluencesunder-standing,actions,anddecisions”. Thisisdifferentfromindividualsconsciouslysuppressingornotsharingtheirbeliefsbecauseothersmayviewtheirbeliefsassociallyundesirable.

??

|23

01.W

hatW

eMeanandWhyItM

atters

Evaluation’sImplicitBias

Evaluation’simplicitbiastowardsahierarchicalormechanisticorientationisalignedwiththewayofthinkingthathasdominatedoursocietysincetheIndustrialRevolutionwhensystemswerethoughtofasstable.Considerthefactorywhereanassemblylineofworkersfitspartstogethertocreateafinalproduct.Themechanisticmodelisalsoreinforcedbybusiness,whichseeksahigh-qualityoutcomethroughefficiencyandeffectiveness.Thehigh-qualityproduct/outcomesbecomestheaimofthework.

There’sawholeschoolofdesignthatcomesoutofthisbusinessmodel.Inthiskindofdesign,peoplegooutintothecommunitytofindoutthepeople’sneeds.Theyaregoodatinvestigatingpeople’slivesinfriendly,non-obtrusiveways.Theygettoknowaculturewiththepurposeofcomingupwithideasaboutproblemsthatcouldbesolvedwithaproductthattheycandevelop.Theydonotidentifyissuesthatcanbesolvedbypeoplesimplyworkingorlivingtogetherdifferentlywithoutmoreproducts.Instead,it’saboutfindingsomethingthatcouldbethebasisofaproductthebusinesscanselltothepeople.Thatmodelofdesignismovingintoevaluationwithoutrecognizingthatthedesignhasanimplicitbiastowardtheproductionofsomething.

Eventhoughevaluationhasapproachessuchasdevelopmentalevaluationandparticipatoryevaluation,evaluators’implicitbiastowardlinearityleadstoviewingnetworkedstructureswithincomplexsystemsasanadd-ontothehierarchicalstructureratherthanasasystemstructureinitsownright.

Evaluationtendstohaveanimplicitbias,unrecognizedbyevaluatorsandevaluationusers,towardassumingsystemschangeinalinearfashion.

Takingacomplexsystemsorientationwillbeoflittleconsequenceunlessweconfrontthepowerandsignificanceoftheimplicitbiaswehavetowardalinearsystemsmodel.

??

|24

01.W

hatW

eMeanandWhyItM

atters

WhyitMattersInsummary,becauseofthetendencyinthefieldofevaluationtofocus

onhierarchicalstructureswithcontrolleddynamicsand/ortobe

unawareofcomplexitysciences,thissectionhasprovidedasetof

vocabularyandconceptsthatareimportantforevaluatorsto

understand.Withoutsuchunderstanding,evaluatorstendtouse

methodologiesthataregroundedinmechanistic(linearmathematical)

modelsthatarecongruentwithhierarchicalstructuresratherthan

developingmodelswherehierarchicalstructuresareentwinedwith

dynamicnetworkstructures.

??

|25

AComplex-SystemsOrientationtoEvaluation

v AComplex-SystemsOrientationtoEvaluation

v EvaluationGroundedinHierarchically-StructuredSystems

v EvaluationGroundedinNetwork-StructuredSystems

v ContrastofEvaluativeFocusinHierarchicalandNetworkedStructures 02

|26

AComplex-SystemsOrientationtoEvaluationAsystemisacollectionofentitiesthatare‘seen’bysomeoneasinteractingtogethertodosomething.Complexsystemsaremassivelyentangledsystems thatincludebothhierarchicalandnetworkedstructures.

Thissectionofthepapershiftsthefocustotakingacomplex-

systemsorientationtoevaluation.Thatis,asevaluators,weaccept

thatweareworkingwithawebofentangledstructureswithmany

gradationsofdynamicsthatrangefromcontrolledandstableto

emergentandflexible.Withinthisdynamicweb,weworkwith

evaluationuserstoidentifythesystemswithinthiswebthatwe

aregoingtoinvestigatethroughtheevaluation.

Recallourearlierdefinitionofasystem:“Asystemisacollectionof

entitiesthatare‘seen’bysomeoneasinteractingtogethertodo

something.”Thus,ourinitialstepinconductingacomplex-systems

orientedevaluationistointeractivelydevelopapicture(literallyor

figuratively)ofthemultiplicityofhierarchicalandnetworked

structuresasseenbyanarrayofstakeholdersrelevanttothe

work.Fromthere,wehighlightandputourfocusonthose

deemedtobethemostimportantstartingpointforthe

evaluation.Wewillnotbediscussingthedetailsofhowtoengage

insuchdeterminationsinthispaper.

|27

02.ACom

plex-SystemsO

rientationtoEvaluation

02.ACom

plex-SystemsO

rientationtoEvaluation

Complex-SystemsOrientedEvaluationDesign

Theevaluatorlooksforthepresenceofthetwobasicsystemstructures—hierarchies(controlled

dynamics)andnetworks(emergentdynamics)—inthesystemsemphasizedintheevaluation.The

evaluatorconsiderstheimplicationsofthestructures andtheirdynamicsfortheevaluation.

(Althoughtherearemanyotheraspectsofcomplexsystemsthatwecouldaddress,thisisourfocus

inthispaper.)

Wealigntheevaluationdesignwiththefundamentalstructures.Asnotedintheabovediscussion,

anygivenevaluationislikelytohaveamixofmethodsandorientation.However,atthispointwe

wanttoclarifythesystemstructuresinwhichtheevaluationisfundamentally“grounded.”

• Istheevaluationgroundedprimarilyinaffectingthehierarchicalstructureswithinwhichthe

interventionislocated?

AND/OR

• Istheevaluationgroundedininfluencingthedynamicnetworksofnodesandconnections?

|28

EvaluationGroundedinHierarchicalStructures

Whenthefundamentalstructureisahierarchicalone,theevaluationemphasisismostlikelygoingtobeonfeaturesofhierarchicalstructuressuchasprograms,policies,andpracticeswithintherelevantsystems.Althoughadaptationofaninterventionmaybeamajorareaofconcernandthesystematicinquiryprocessitselfmayconsumethemajoramountoftime,theevaluative aspectislikelytoemphasize:

v Measuringoutcomesv Linkingoutcomesandactivitiesv Applyingpredeterminedevaluationcriteriav Buildingevaluationcapacityaroundtechnicalaspectsofdeterminingoutcomesandevaluation

criteria

|29

02.ACom

plex-SystemsO

rientationtoEvaluation

EvaluationGroundedinNetworkedStructures

Whenthefundamentalstructureisanetwork,theevaluationfocusesonthefundamentalfeaturesofnetworks—nodesandconnections.Thesenodesandconnectionsmaywellincludehierarchicalfeaturesbuttheemphasisisonthenetworkedaspects.Thustheevaluative aspectislikelytoemphasize:

v Mappingpatternsofmovementandchangerelatedtovaluesv Facilitatingconversationsaboutmerit,worth,andsignificancev Buildingthecapacityofnetworkstosustaincontextuallyappropriateattentiontomerit,worth,and

significanceamongmultiplestakeholders

|30

02.ACom

plex-SystemsO

rientationtoEvaluation

Neighborhood

Faith-BasedOrganization

SocialMovement

GovernmentalAgency

NationalCommunityLeadershipNetwork

ExampleofEvaluationGroundedinNetworked-StructuresAssumethattheinterventionofinterestisanationalnetworktosupportacultureofhealth.Iftheevaluationistobegroundedinnetworkedstructures,theevaluationmayfocusonfollowingvariousmembersofthenationalnetworkastheyworkthroughnetworksintheircommunitylocation.Thediagrambelowillustratesthenetworksofonememberofanationalnetwork.Thenetworkontheleftisaportionofthelocalcommunity;onememberofthenetworkisbeingfollowedintohercommunity.Therestofthediagramshowsthatpersoninherlocalnetworks.

|31

02.ACom

plex-SystemsO

rientationtoEvaluation

Whentheevaluatorofthenationalnetworkconductsanevaluativefacilitationandinquiryinthisnetworkmember’scommunity,shemightfocusherevaluationonequity,leadership,andpowerdifferentialsinthevariousstructuresinthecommunityasillustratedbelow.

Neighborhood

Faith-BasedOrganization

SocialMovement

GovernmentalAgency

NationalCommunityLeadershipNetwork

Inquiryfocusesonunderstandingdifferencesinthe

powerofhierarchicaland

networkedstructures.

ExampleofEvaluationGroundedinNetworked-Structures

|32

02.ACom

plex-SystemsO

rientationtoEvaluation

v Mappatternsofmovementandchangerelatedtovalues

v Facilitateconversationsaboutmerit,worth,andsignificance

v Buildcapacityofnetworkstosustaincontextuallyappropriateattentiontomerit,worth,andsignificance

HierarchicalStructures NetworkedStructures

v Measureoutcomesv Linkoutcomesandactivitiesv Applyevaluationcriteriav Buildevaluationcapacityaboutaspectsofsystematicinquiry

ContrastofEvaluativeFocusinHierarchicalandNetworkedStructures

|33

02.ACom

plex-SystemsO

rientationtoEvaluation

AComplex-SystemsOrientationtoEvaluationthatSupportsa

CultureofHealthv Culturev ACultureofHealthv ACultureofHealthActionFrameworkv ExampleofanEvaluationSituationandApproach

§ ExampleEvaluationBackground§ ExampleEvaluationApproach§ ExampleEvaluationQuestions:Interconnections,Boundaries,

Perspectives 03|34

CultureCultureisdefinedinanumberofways.Herearethree:

• Cultureisasharedsetofvalues,customs,andbeliefscommontoaparticulargrouporsocietythatproducespatternsofbehaviorsandexperiences.

• Cultureasmultiplediscourses,occasionallycomingtogetherinlargesystemicconfiguration,butmoreoftencoexistingwithindynamicfieldsofinteractionandconflict.(Dirks,Eley,&Ortner,1994,p.4)

• Incontrastwiththeclassicview,whichpositscultureasaself-containedwholemadeupofcoherentpatterns,culturecanarguablybeconceivedasamoreporousarrayofintersectionswheredistinctprocessescrisscrossfromwithinandbeyonditsborders.(Rosaldo,1993,p.20)

Allsaid,cultureisacomplexsystemwith massivelyentangledentitiesandhierarchicalandnetworkedstructures.Itsboundariesarepermeableanditisdispersedthroughsociety.Someculturalaspectsareimposed byoutsidegroupsorindividuals whileothersarisefromwithinthegroup.

Cultureisoftenthoughtofintermsofracialorethnicgroups.Cultureemergesinmanygroups.Abusiness,neighborhood,community,sportsteam,family,ornetworkdevelopsitscultureovertime.Thesharedvalues,customs,andbeliefsmayormaynotbeexplicitandamemberofthegroupmaybeconsciousorunconsciousoftheculture.

Culture isacomplexsystemwithmassivelyentangledentitiesandhierarchicalandnetworkedstructures.

|35

03.ACom

plex-SystemsO

rientationtoEvaluationthatSup

portsa

CultureofH

ealth

03.ACom

plex-SystemsO

rientationtoEvaluationthatSup

portsa

CultureofH

ealth

ACultureofHealthTheaimistofocusonbuildingacultureofhealthratherthan

focusingonlyonhealthcareservices.

Toaccomplishthis,changesmustbemade

inbeliefsandassumptions,andthe

structuresofsocialsystems.

“Recognizethathealthandwell-beingcanbegreatlyinfluencedbycomplexsocialfactors:wherewelive,howwework,thesoundnessandsafetyofoursurroundings,andthestrengthandresilienceofourfamiliesandcommunities”*

“ComprehensiveCultureofHealth…enablesallinourdiversesocietytoleadhealthierlives,nowandforgenerationstocome”*

“Creatingasocietythatgivesallindividualsanequalopportunitytolivethehealthiestlivespossible,whatevertheirethnic,geographic,racial,socioeconomic,orphysicalcircumstanceshappentobe”*

RWJFisfocusedonbuildingacultureofhealththatemphasizesequityandanunderstandingthathealthisdeterminedbymanysocialconditions.BylabelingtheirworkasCulture ofHealth,RWJFrecognizestheimportanceofhealthbeingbasedonsharedvalues,customs,andbeliefsandhowtheseneedtochangewithinvariousgroupstohavepopulationhealth,well-being,andequity.RWJF’sapproachcontraststoonethatfocusesonlyonhealth-careservices.Thehealth-care-serviceapproachassumesthatindividualsarelargelyresponsible fortheirownhealth;itignorestheinequitiesandtheconfusingprocessesandmultipleorientationswithinhealthcareandothersocialdeterminantsofhealth.RWJF’sapproachfocusesonpeoplechangingtheirbeliefsandassumptionsandthestructuresoftheirsocialsystems,i.e.,changingtheculturetoaCultureofHealth.

*Copyright2015RobertWoodJohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth.

|36

ACultureofHealthActionFrameworkBasedonthesegeneralbeliefsandassumptions,theFoundationhasidentifiedfourareasforactiontobringaboutthecultureshift.Thefoundationhasalsostatedanencompassingoutcomethatwouldresultfromthisshift.

Notethattheframeworkisbroad.Itdoesnotgivespecificactionsforpeopletotakebutprovidestheareasonwhichtofocusandincorporatesthebeliefsthatarekeytothenewculture.Itisgivingpeopleatanyplaceinsocietyareasinwhichtofocus.

Buildinganewculturemaybelessaboutactionswedoandmoreaboutadialoguethatflowsthroughandamongus.

Imagecopyright2015RobertWoodJohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth. |37

03.ACom

plex-SystemsO

rientationtoEvaluationthatSup

portsa

CultureofH

ealth

ExampleofanEvaluationSituationandApproach Acoalitionthatrecentlyreceivedagranthasrequestedsuggestionsabout

conductingitsevaluationtodetermineifthemanysystemsinvolvedintheirworkareactuallyfunctioninginawaythatchangesthecultureregardinghealth.TheyareusingtheRWJFCultureofHealthActionFrameworkastheirguide.

Theexampleprovidessuggestionsrelatedtothegeneralevaluationapproachandevaluationquestions.

Here’sthegranteesituation:

Overa20-yearperiod,acoalitionhasguidedthelocatingofawiderangeofhealth-careservicesinonehealth-carecomplex.Thecoalition,locatedinalow-incomeregionofasouthwesternstate,includesvariousentitiessuchasnonprofitorganizations,health-careserviceproviders,andresidentsfromtheurbanneighborhoodsandruralareas.Severalyearsago,withcommunityinput,thecoalitiondevelopedastatementofsharedvalues.

Acommunitysurveyrevealedaneedforaccesstomorehealthyfood.So,thecoalitionisnowplanningtoaddagrocerystoreinthehealth-carecomplex.

Thecoalitionmembershaveanunderstandingofcomplexsystemsfromarecentroundofseminars.Theyareparticularlyinterestedinthenodesandconnectionsofthenetworksandwebsoftheircommunityandwhetherthey,asmembersofacoalition,areanimportantnodeinbuildinganeconomically-equitablecommunity.Theyarealsoawarethatcommunityorganizerswhoarenewmembersofthecoalitionthinkthatthecommunitysurveyprovidedaninadequateunderstandingofthecommunityasawhole.Photos(exceptfarmworkers)copyright2015RobertWood

JohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth;photosbyTyroneTurner.Farmworkers’photosretrievedfrompublicdomain. |38

03.ACom

plex-SystemsO

rientationtoEvaluationthatSup

portsa

CultureofH

ealth

ExampleEvaluationSituation

Coalitionmembersarepleasedwiththeprogresstheyhavemadeonoverallindicatorsofpopulationwell-beingandhealth,buttheyarenotsureifdeepchangesintheculture(changesinvalues,beliefs,andassumptions)areoccurringthatwillsupportasustainablecultureofhealthasdescribedbytheRWJFCultureofHealthactionframework.

Thecoalitionhasworkedcollaborativelyinthehealthcomplextobuildawidespreadunderstandingofthecentralityofhealthinthegovernmentalagencies,businesses,schools,anduniversities.Health-careserviceprovidersareintegratingtheirtechnologicalsystemstosharedataandmakeiteasierforresidentstoworkwithmultipleproviders.Theynowrecognizethattheseeffortsareallfocusedonhierarchicalstructures.

Theyareuneasyaboutwhethersufficientattentionisbeinggiventobuildinganeconomically-equitablecommunity,astructurethatismoregroundedinnetworks.

Intheirnextphaseofevaluationtheywanttofocusspecificallyonthefunctioningoftheircoalitionasanodewithinthemanyinterconnectednetworksandwebsoftheircommunitythatareespeciallyrelevanttobuildinganeconomically- equitablecommunity.

Photos(exceptfarmworkers)copyright2015RobertWoodJohnsonFoundation/FromVisiontoAction:AFrameworkandMeasurestoMobilizeaCultureofHealth;photosbyTyroneTurner.Farmworkers’photosretrievedfrompublicdomain.

|39

03.ACom

plex-SystemsO

rientationtoEvaluationthatSup

portsa

CultureofH

ealth

Workingwithasystems-orientedevaluatorfromanonprofitorganizationthatisamemberofthecoalition,thecoalitionmembersbeganbyidentifyingtheseguidelinesforconductingasystems-orientedandnetwork-focusedevaluation.

• Usebroadlydefinedboundaries fordeterminingwhattypesofnetworksandwebsexistinthecommunitythatmayberelevant.

• Attendtothe interconnections (especiallypowerrelationships) amongthefullrangeofstakeholders(individualsandcollectiveentities)inthesituation.

• Lookforpatternsofinterconnectednodes;lookformanytypesofresources;andlookathowthesevariousresourcesflowthroughthenetworkandareexchangedinthenodesofthenetworks.

• Encouragetransparencyofperspectives (values,beliefs,andassumptions)inallphasesoftheevaluation.

• Emphasizefacilitationthatdeterminesmerit,worth,andsignificancethroughouttheprocessasitrelatestohealthforall.

• Focusonaskingthe“right”questionsforpurposesofsharedreflectionandinquiryratherthannecessarilyansweringthequestionsinafinalsense.

• Emphasizebuildingthecapacityofcoalitionmemberstoengageinevaluative,reflectiveinquirythathelpsthemexpendtheleastefforttobuildtheself-organizingcapacityofthenetworkstosupportaneconomically-equitablecommunity.

• Drawonexpertisefrommultipledisciplinesrelevanttothesituation.

ExampleEvaluationApproach

|40

03.ACom

plex-SystemsO

rientationtoEvaluationthatSup

portsa

CultureofH

ealth

Herearesomeofthequestionstheyareconsidering.Theyasksomequestionstounderstandthecurrentandpastinteractionsabouttheeconomicallyequitablenatureoftheircommunityandhowtheestablishmentofagrocerystoreinthehealthcarecomplexmightrelatetothisdesiredcondition.Theyaskotherquestionstounderstandthebeliefs,assumptions,perspectives,andperceptionsaboutthemeaningandnatureofaneconomicallyequitablecommunity.

Interconnections

• Whatpatternsofpowerrelationshipsarepresentandhowaretheyshifting?

• Whatistheebbandflowofchangesbetweentheformalhealthorganizationsandtheresidentsofthecommunity?

• Whatshiftsareoccurringintheflowofinformation?

Boundaries

• Whatboundarieswithinandacrossorganizationsandwiththecommunityanditsneighborhoodgroupsareshifting?

• Whatdifferencesareevidentinthepermeabilityoftheboundarieswithinandacrossorganizationsandwiththecommunityanditsneighborhoods?

• Whatshiftsareoccurringinwhoisinvolvedandinwhatwaysinvarioustypesofdecision-making?

• Howarepatternsofactionandthoughtshiftingregardinghealthacrossgenerations?Whoisincluded?Whoisnot?

Perspectives

• Whatshiftsinvalues,beliefs,andassumptionsareoccurringrelatedtoequityanddemocracy?

• Whatshiftsinvalues,beliefs,andassumptionsareoccurringregardingtheimportanceofhealth?

• Whatshiftsinvalues,beliefs,andassumptionsareoccurringregardingwhatconstituteshealth?

ExampleEvaluationQuestions:Interconnections,Boundaries,andPerspectives

|41

03.ACom

plex-SystemsO

rientationtoEvaluationthatSup

portsa

CultureofH

ealth

CalltoAction

v ImplicationsforEvaluationandEvaluators

04|42

04.CalltoActio

n

ImplicationsforEvaluationandEvaluatorsThispapercallsonevaluatorsandevaluationuserstoattendtotwofundamentalstructuresofcomplex

socialsystems—hierarchiesandnetworks.Asastarter,considerthefollowing:

• Engageinevaluationwithattentiontobothhierarchicalstructureswithcontrolleddynamicsand

networkedstructureswithemergentdynamics.

• Recognizethatevaluationcapacitybuildingislikelytolookdifferentwhenworkingwithhierarchical

structuresversusnetworkedstructures.

• Payspecialattentiontoevaluativefacilitationaswellasinquiryprocesses.

• Recognizethatoutcomesinahierarchicalstructurearelikelytobedefinableproducts,knowledge,or

services,whereasoutcomesinanetworkedstructurearelikelytobechangesinpatternsofactionand

thinkingacrosstimeandlocations.

• Recognizethedifferenceinrolesofthetwotypesofstructuresinsustaining,restoring,andregenerating

society.

• Engageinconversationswithcolleaguesabouttheapplicationofthesesystemsconceptsandsharewith

uswhatyouarelearning.

|43

Resources

v Referencesv Acknowledgements

|44

ReferencesCapra,F.&Luisi,P.(2014).Thesystemsviewoflife:Aunifyingvision.Cambridge,MA:CambridgeUniversityPress.Dirks,N.B.,Eley,G.,&Ortner,S.B.(1994).Introduction.InN.B.Dirks,G.Eley,&S.B.Ortner(Eds.),Culture/power/history:Areaderin

contemporarysocialtheory(pp.3-45).Princeton,NJ:PrincetonUniversityPress.Eoyang,G.(1997).Conditionsforself-organizinginhumansystems.Unpublisheddoctoraldissertation,UnionInstituteandUniversity,

Cincinnati,OH.Fairclough,N.andWodak,R.,(1998)."CriticalDiscourseAnalysis"inDijk,T.A.(Ed.)(1998)DiscourseasSocialInteraction:Discoursestudies:A

multidisciplinaryIntroduction(Vol.2).LondonSAGE.Girard,JohnP.&Girard,JoAnnL.(2015)."Definingknowledgemanagement:Towardanappliedcompendium"(PDF). OnlineJournalofApplied

KnowledgeManagement.3 (1):1–20.Jackson,M.C.(2003).Systemsthinking:Creativeholismformanagers.WestSussex,England:JohnWileyandSonsLtd.KelloggFoundation,W.K.(2007).Designinginitiativeevaluation:Asystems-orientedframeworkforevaluatingsocialchangeefforts. Battle

Creek,MI:W.K.KelloggFoundation.Mathison,S.(2005).Encyclopediaofevaluation.ThousandOaks,CA:SAGEPublications.Meadows,D.H.(2008).Thinkinginsystems.WhiteRiverJunction,VT:ChelseaGreenPublishing.OpenUniversity.(2011).Systemsthinkingandpractice.Retrievedfromhttp://www.open.edu/openlearn/science-maths-

technology/computing-andict/systems-computer/systems-thinking-and-practice/content-section-0Parsons,B.(2012).Usingcomplexityscienceconceptswhendesigningsysteminterventionsandevaluations(3rded.).Retrievedfrom

http://insites.org/resources/using-complexity-science-concepts-when-designing- system-interventions-and-evaluations/Ramage,M.,&Shipp,K.(2009).Systemsthinkers.London,England:Springer.Rosaldo,R.(1993).Culture&truth:Theremakingofsocialanalysis.Boston,MA:BeaconPress.Scriven,M.(2013).Thefoundationandfutureofevaluation.InThefutureofevaluationinsociety:AtributetoMichaelScriven.Donaldson,S.

(Ed.).Charlotte,NC:InformationAgePublishing,Inc.Staats,C.,Capatosto,K.,&Contractor,D.(2015).Stateofthescience:ImplicitBiasReview2015. Columbus,Ohio:KilwanInstitutefortheStudy

ofRaceandEthnicity.Williams,B.&Hummelbrunner,R.(2009).Systemconceptsinaction:Apractitioners’toolkit. Stanford,CA:StanfordUniversityPress.Zimmerman,B.,Lindberg,C.,&Plsek,P.(2001).Edgeware:Insightsfromcomplexityscienceforhealthcareleaders. Irving,TX:VHA.

Websiteshttp://www.insites.org/

http://www.fsg.org/http://www.evaluationinnovation.org/

https://www.aspeninstitute.org/

Resources

|45

AcknowledgementsBeverlyParsons,PatriciaJessup,andMarahMoorewrotethispaperwithsupport

fromtheRobertWoodJohnsonFoundation.Theviewsexpressedheredonot

necessarilyreflecttheviewsoftheFoundation.

WewishtoexpressourthankstoLauraLevitonoftheRobertWoodJohnson

Foundationforherinsights,review,andfeedbackthroughoutthecompletionof

thegrant.

OurappreciationisalsoexpressedtoourcolleaguesintheAmericanEvaluation

Association’sSystemsinEvaluationTopicalInterestGroupforthemany

conversationsthathaveinformedthethinkingreflectedinthispaper.

WearealsodeeplygratefultoRosemaryReinhartforhereditingandKathy

Wyckoffforherformattinganddesignofthepaper.

Resources

|46October2016