6/13/16

1

UsingSETtoEvaluateaCardiacSystemofCareRenger- CES2016

TheChallenge

6/13/16

2

SystemsThinking,Systems,andEvaluation• Whatisyourunderstandingofsystemsthinkingandhowitsbeingusedinevaluation?

• Therewasnoevaluationtheoryforevaluatingsystems.

SystemEvaluationTheory(SET)

• CatherineCourage:Innovative-HiredaBiologist• Groundedinsystemtheory.

• GoalofSETistoevaluatesystemefficiencyandeffectiveness.• CardiacCareEfficiency=time*• CardiacCareEffectiveness=improvedpatientoutcomes

6/13/16

3

Whatdoyouneedtoknowbeforeyoucanproceedtoevaluatesystemefficiency?

• Needtodefinethesystem.• Why?

• Youneed toknowhowthesystemoperatesbeforeyoucandevelopanevaluationstrategytodeterminewhetheritisoperatingefficientlyandeffectively.• Establishesscopeandbudget.

• Whatisthedefinition ofasystem?• Parts,componentsworkingtogethertowardacommongoal.

SETGuidingPrinciple1:DefineSystem

• Startbydefining boundaries

6/13/16

4

9-1-1Dispatch

LargerUrbanHospital

SETGuidingPrinciple1:DefineSystem

• Startbydefining boundaries• Definesubsystems

6/13/16

5

9-1-1Dispatch

SmallRuralHospital

EmergencyMedicalServices

LargerUrbanHospital

SETGuidingPrinciple1:DefineSystem

• Startbydefining boundaries• Definesubsystems• Definesubsystemboundaries

6/13/16

6

9-1-1Dispatch

SmallRuralHospital

EmergencyMedicalServices

LargerUrbanHospital

StartStart

Start

Start

End End End

End

SETGuidingPrinciple1:DefineSystem

• Startbydefining boundaries• Definesubsystems• Definesubsystemboundaries• Makewithin subsystemoperatingproceduresandprocesses explicit

6/13/16

7

9-1-1Dispatch

SmallRuralHospital

EmergencyMedicalServices

LargerUrbanHospital

StartStart

Start

Start

End End End

End

Withinsubsystemprocesses

SETGuidingPrinciple1:DefineSystem

• Startbydefining boundaries• Defineidentify subsystems• Definesubsystemboundaries• Makewithin subsystemoperatingproceduresandprocesses explicit• Makebetween subsystemoperatingproceduresandprocesses explicit

6/13/16

8

9-1-1Dispatch

SmallRuralHospital

EmergencyMedicalServices

LargerUrbanHospital

StartStart

Start

Start

End End End

End

Betweensubsystemprocesses

Betweensubsystemprocesses

Betweensubsystemprocesses

Howwouldyoudetailsystemprocessesandprocedures?

• DetailedProcessFlowMap• Methodwillbepublished innextvolumeofCPJE.• PFMprocessechoes CatherineCourageemphasis onimmersion inclientcontext.

6/13/16

9

WhataresomeadditionalbenefitsofProcessFlowMapping?

• Abilitytomakemeaningful recommendations• 60%ofrecommendation reportsresulted insystemchangewithin oneweekofreleasingthereport.

• Trust,thenleadingtosharingofotherimportantinformation• Falsifyingpatientrecords.

6/13/16

10

SETPrinciple2:EvaluateSystemEfficiency

• Whydoweneedtoevaluatesystemefficiencyfirst?

SET:Factorsaffectingsystemefficiency

• Systemattributes• Leadership.How?• Informationtechnology. How?

• Competent/capablesystemactors.How?

• Systemfeedbackmechanisms*• Subsystem interactions*

6/13/16

11

9-1-1Dispatch

SmallRuralHospital

EmergencyMedicalServices

LargerUrbanHospital

StartStart

Start

Start

End End End

End

SystemFeedbackMechanisms

Whatcharacteristicsofeffectivesystemfeedbackneedtobeevaluated?

6/13/16

12

Howdoyouevaluatesystemfeedbackmechanisms?

• Examinewhethersystemismonitoringefficiency.• Ifthereisn’tany- then thatisthefirstrecommendation.

• Ifthereisasystemfeedbackmechanism, thenevaluatewhetherthecharacteristicsforeffectivefeedbackarebeingmet.• Qualitative

• Interviewswithleadershipandusers(oftensharedduringPFMprocess)• Quantitative

• Analysesofdatabasesformeaningful dataelements,validityofdata(Dr.Qinanalysis)

Whatdidwelearnaboutfeedbackmechanisms?

• Notspecific• Explicitlystatedprocesseslacking• Dataneeded forCQInotclear

• Notcredible• Dataentry integrity• Urbanbiasedbenchmarks:CARES

• Nottimely• Severalmonthstogetfeedback• Oftenloopnotclosedatall

• Notsufficiently frequent

6/13/16

13

Systemchangesresultingfromtheevaluationoffeedbackmechanisms

Evaluatingefficiency:Subsysteminteractions

• Howdoyoulocateinteractionstoevaluate?• Documentedprocessesandprocedures

• DuringPFMyouneedtobeawareoftheneedtounderstand anddocumenthowsubsystemscommunicate witheachother.

6/13/16

14

9-1-1Dispatch

SmallRuralHospital

EmergencyMedicalServices

LargerUrbanHospital

StartStart

Start

Start

End End End

End

PossibleCascadingFailures

PossibleCascadingFailures

PossibleCascadingFailures

How doyouevaluatesystemforpossiblecascadingfailures?

6/13/16

15

Systemchangesresultingfromevaluationofcascadingfailures

Summary

• SETisverypractical• Dependingonsizeofsystem;evaluationcouldbeexpensive;morethanwhatclientsareexpectingbasedonprogramevaluationexperience• Somesystemprinciplesmirrorprogramevaluation(e.g.,efficiency =processimplementation; effectiveness =outcomes)• Mostsystemprinciplesrequiredifferentthinking

• SEThelpsdirectthisthinking• Manyofourevaluationmethodscanbeusedtoevaluatesystems• ThepowerofSETisspreading

• Lateadoptingstateswanttomovetosystemsevaluation• Australiananti-corruption

6/13/16

16

RalphRenger,PhDralph.renger@med.und.edu520-203-1038

EvaluatingEffectiveness

• Systemprinciples ofinterdependence, holistic,&synergycomeintoplay

• Makesnosense toengageincause-effectdesigns toevaluateasystem• Can’tisolateeffectofonesystempart• Isolationcontradictsthesystemapproach

6/13/16

17

INFO

RMATIONTECHNOLO

GY

COMPETENT&CAPABLEACTORS

BoundariesWithinSubsystemProcessesBetween SubsystemProcessesWithinSubsystemFeedbackMechanismsBetween subsystemFeedbackMechanismsInputsAttributes(CAPS)

Legend

Subsystem1

Start

End

Subsystem2

Start

End

Subsystem3

Start

End

Subsystem4

Start

End