Michael Quinn Patton May, 2008 Evaluating the Complex: Getting to Maybe Michael Quinn Patton Oslo, Norway 29 May, 2008

  • View
    214

  • Download
    2

Embed Size (px)

Transcript

  • Slide 1
  • Michael Quinn Patton May, 2008 Evaluating the Complex: Getting to Maybe Michael Quinn Patton Oslo, Norway 29 May, 2008
  • Slide 2
  • Michael Quinn Patton May, 2008 Evolving Understandings I keep changing what I said. Any person who is intellectually alive changes his ideas. If anyone at a university is teaching the same thing they were teaching five years ago, either the field is dead, or they havent been thinking. Noam Chomsky The Professor Provaocateur, The New York Times Magazine, Nov. 2, 2003: 13.
  • Slide 3
  • Michael Quinn Patton May, 2008 Interpretive Frameworks May 2003 Harvard Business Review "The High Cost of Accuracy." Kathleen Sutcliffe and Klaus Weber. They concluded that "the way senior executives interpret their business environment is more important for performance than how accurately they know their environment."
  • Slide 4
  • Michael Quinn Patton May, 2008 They further concluded that it is a waste of resources to spend a lot of money increasing the marginal accuracy of data available to senior executives compared to the value of enhancing their capacity to interpret whatever data they have. Executives were more limited by a lack of capacity to make sense of data than by inadequate or inaccurate data. In essence, they found that interpretive capacity, or "mind-sets," distinguish high- performance more than data quality and accuracy.
  • Slide 5
  • Michael Quinn Patton May, 2008 Evaluations Traditional Interpretive Framework In the beginning
  • Slide 6
  • Michael Quinn Patton May, 2008 Original Primary Options Formative and Summative Evaluation (Mid-term and End-of-Project Reviews)
  • Slide 7
  • Michael Quinn Patton May, 2008 Evidence-based Practice Evaluation grew up in the projects testing models under a theory of change that pilot testing would lead to proven models that could be disseminated and taken to scale: The search for best practices and evidenced-based practices
  • Slide 8
  • Michael Quinn Patton May, 2008 Fundamental Issue: How the World Is Changed Top-down dissemination of proven models versus Bottoms-up adaptive management
  • Slide 9
  • Michael Quinn Patton May, 2008 Models vs. Principles Identifying proven principles for adaptive management (bottoms-up approach) versus Identifying and disseminating proven models (top down approach)
  • Slide 10
  • Michael Quinn Patton May, 2008 Conditions that challenge traditional model-testing evaluation High innovation Development High uncertainty Dynamic Emergent Systems Change Adaptive Management
  • Slide 11
  • Michael Quinn Patton May, 2008 Mintzberg on Strategy Two types of strategy: Intended & Emergent Unrealized Strategy Intended Strategy Deliberate Strategy Realized EmergentStrategy Strategy
  • Slide 12
  • Michael Quinn Patton May, 2008 Re-conceptualizing Use Use is a process not a event Use involves an interaction not just a report Use involves training for use not just delivery of results
  • Slide 13
  • Michael Quinn Patton May, 2008 Some premises : Evaluation is part of initial program design, including conceptualizing the theory of change Evaluators role is to help users clarify their purpose, hoped-for results, and change model. Evaluators can/should offer conceptual and methodological options. Evaluators can help by questioning assumptions. Evaluators can play a key role in facilitating evaluative thinking all along the way. Interpretative dialogue is critical. Designs can be emergent and flexible.
  • Slide 14
  • Michael Quinn Patton May, 2008 Three ways of conceptualizing and mapping theories of change Linear Newtonian causality Interdependent systems relationships Complex nonlinear dynamics
  • Slide 15
  • Michael Quinn Patton May, 2008 Linear Logic Model INPUTS (people, materials) ACTIVITIES (processes) OUTPUTS OUTCOMES CHANGES IN PEOPLES LIVES IMPACTS CHANGES IN COMMUNITIES
  • Slide 16
  • Michael Quinn Patton May, 2008 Pushing Force (Non-directional) Dissatisfaction with the Status Quo (Inertia) a.k.a. Cost of Change Resistance to Change Pulling Force (Directional) a.k.a. Desirability of the end state Compelling Vision
  • Slide 17
  • Michael Quinn Patton May, 2008 Pushing Force (Non-directional) Dissatisfaction with the Status Quo (Inertia) a.k.a. Cost of Change Resistance to Change Pulling Force (Directional) a.k.a. Desirability of the end state Compelling Vision First StepsBelievability Causal Diagram of Beckhards Change Formula
  • Slide 18
  • Michael Quinn Patton May, 2008 Feedback
  • Slide 19
  • Michael Quinn Patton May, 2008
  • Slide 20
  • Step Two: THEN A MIRACLE OCCURS I think you should be more explicit here in step two.
  • Slide 21
  • Michael Quinn Patton May, 2008 Systems Parts are interdependent such a change in one part changes all parts The whole is greater than the sum of the parts Focus on interconnected relationships Systems are made up of sub-systems and function within larger systems
  • Slide 22
  • Michael Quinn Patton May, 2008 Systems Concepts in Evaluation An Expert Anthology. 2006. Bob Williams and Iraj Imam AEA Monograph, EdgePress/AEA Point Reyes CA.
  • Slide 23
  • Michael Quinn Patton May, 2008
  • Slide 24
  • Understanding the Elephant from a Systems Perspective
  • Slide 25
  • Michael Quinn Patton May, 2008 The relationship between what goes in and what comes out What conceptual framework informs front-end evaluation work?
  • Slide 26
  • Michael Quinn Patton May, 2008 Teen Pregnancy Program Example
  • Slide 27
  • Michael Quinn Patton May, 2008 Logic Model for Pregnant Teens Program 1. Program reaches out to pregnant teens 2. Pregnant teens enter and attend the program (participation) 3. Teens learn prenatal nutrition and self-care (increased knowledge) 4. Teens develop commitment to take care of themselves and their babies (attitude change) 5. Teens adopt healthy behaviors: no smoking, no drinking, attend prenatal clinic, eat properly (behavior change) 6. Teens have healthy babies (desired outcome)
  • Slide 28
  • Michael Quinn Patton May, 2008 Systems web showing possible influence linkages to a pregnant teenager Teachers/ other adults Young pregnant woman's attitudes & behaviors Her parents & other family members Child's father & peers Prenatal program staff Her peer group
  • Slide 29
  • Michael Quinn Patton May, 2008 Program systems web showing possible institutional influences affecting pregnant teenagers: SCHOOL SYSTEM Young pregnant women's attitudes & behaviors Prenatal Clinic and Hospital Outreach Church Prenatal program Other community- based youth programs Other Systems -- welfare -- legal -- nutrition programs -- transportation -- child protection -- media messages Context factors -- politics -- economic incentives -- social norms -- culture -- music Youth Culture
  • Slide 30
  • Michael Quinn Patton May, 2008 Using Different System Lenses to Understand a particular System Biologic System Emergence Coordination/synergy Structure, Process, Pattern Vitality Sociologic System Relationships Conversations Interdependence Loose-tight coupling Meaning/sense Mechanical / Physical System Flow Temporal Sequencing Spatial Proximities Logistics Information Economic System Inputs/Outputs Cost/Waste/Value/Benefits Customers/Suppliers Political System Power Governance Citizenship Equity Anthropologic System Values Culture/Milieu Information System Access Speed Fidelity/utility Privacy/security Storage Psychological System Organizing Forces Field Ecological/Behaviour Settings SYSTEM DIMENSIONS
  • Slide 31
  • Michael Quinn Patton May, 2008
  • Slide 32
  • Complex Dynamic Systems Configuration
  • Slide 33
  • Michael Quinn Patton May, 2008 HIV/AIDS Example Hits every system: health, family, social, religious, economic, political, community, international Requires multiple interventions on multiple fronts in all subsystems simultaneously Resulting reactions, interactions, consequences dynamic, unpredictable, emergent, and ever changing
  • Slide 34
  • Michael Quinn Patton May, 2008 Challenges: Situation Recognition and Appropriate Evaluation Designs
  • Slide 35
  • Michael Quinn Patton May, 2008 The nature of EXPERTISE
  • Slide 36
  • Michael Quinn Patton May, 2008 Contingency-based Evaluation Situational analysis & responsiveness Context sensitivity Clarify and focus on intended users: stakeholder analysis Clarify and focus on intended uses Methodological appropriateness Criteria for evaluating the evaluation: credibility, meaningfulness
  • Slide 37
  • Michael Quinn Patton May, 2008 Seeing Through A Complexity Lens You don't see something until you have the right metaphor to let you perceive it. Thomas Kuhn
  • Slide 38
  • Michael Quinn Patton May, 2008 Complex Nonlinear Dynamics Nonlinear: Small actions can have large reactions. The Butterfly Wings Metaphor Emergent: Self-organizing, Attractors Dynamic: Interactions within, between, and among subsystems and parts within systems can volatil