IMPAQ INTERNATIONAL LLC. PRESENTER: ANNE ......AEA 2016 – DESIGN THINKING: EVALUATION & BEHAVIORAL SCIENCE DESIGN THINKING Learning from the design process. Plan Do Study Act The

IMPAQ INTERNATIONAL LLC.

PRESENTER: ANNE CHAMBERLAIN

DESIGN THINKING AS FRAMEWORK FOR EVALUATION + BEHAVIORAL SCIENCE COLLABORATION

AMERICAN EVALUATION ASSOCIATION

AEA 2016 – DESIGN THINKING: EVALUATION & BEHAVIORAL SCIENCE

OVERVIEW▸ Design Thinking and Human-Centered Design

▸ How Should We Work Together?

▸ Where They Don’t

▸ Where Program Evaluation and Behavioral Science Intersect

▸ Behavioral Science


DESIGN THINKING▸ Learning from the

design process.Plan

Do

Study

Act

▸ The systematic (if not explicit) decision-making that yields “a design.”


HUMAN-CENTERED DESIGNA process that yields products or processes that are made in response to human behavior and decision-making.

Human Behavior

Products

Processes

Programs


DESIGN THINKING + HUMAN-CENTERED DESIGN

▸ Focus on how decisions are made

▸ Draw heavily from behavioral science


DESIGN THINKING + HUMAN-CENTERED DESIGN

Behavioral Science ▸ Focus on how

decisions are made

▸ Draw heavily from behavioral science


BEHAVIORAL SCIENCE

▸ Rigorous, iterative testing

▸ Use of evidence-based changes to fix program snags

▸ Quantitative evidence of patterns in behavior (decisions)


BEHAVIORAL SCIENCE

▸ Energy consumption report-cards

▸ 401-k opt-out default


NOTHING NEW, PER SE▸ Graphic designers,

instructional designers…

▸ Anthropologists, sociologists…

.41%3.15%

6.7%12.18%

21.83%

55.74%

0%

10%

20%

30%

40%

50%

60%

EducatorStaff PathologistDirector or Administrator


WHY NOW

▸ Behavioral science is in the RFPs

▸ Behavioral scientists have the ear of policymakers


ContractPurpose Evaluand RoleofBS BSRequirements ACF:

TANFExploratory ExistingTANFoffices

thathavechangedorgculture&exemplars

Informapproachtoresearch

“knowledgewillbehelpful”

DOL:X-Progrm

Improveprogramperformance

Small,BS-informedinnovations

• “Intervention”• Investigativeapproach

PI:5yearsapplyingsocial&BSprinciplestosocialwelfareprograms.PhDinBSoranotherrelevantfield.

DOL:VETS

• IdentifyrelevantBSprinciples&findingsforTAPprogram

• Improveprogramperformance

• Impactstudy

• TAPSprogram(QED)• Smallenhancements

• Programenhancements(e.g.,intervention)

• TWG• PI:5yearsapplyingsocial&BSprinciplesinempl/trainingprogs.PhDinBSoranotherrelevantfield.

ED:IES Impactofparentmessagingstrategies

Evaluatorwillhelpdeterminemessagefrequency,sequencing,content

• “Intervention”• Investigativeapproach

TWG

BEHAVIORAL SCIENCE


INTERSECTIONS: PURPOSE

▸ To improve social programs • Responsibilities for General Public

Welfare ▸ To produce positive

social change

• Fostering Public Understanding

• Service in the Public Interest

• Enhancement of the Public Good


INTERSECTIONS: TRAINING & TOOLS

▸ Research

• Surveys

• Interviews

• Observations


DIFFERENCES: ROOTS AND PERSPECTIVE

▸ Anthropology

▸ Sociology

▸ Public Policy

• Maths

• Business

• Psychology

▸ Social Justice

▸ Cultural Responsiveness

• “Hard Science,” Empirical Research

• Efficiencies


DIFFERENCES: ENTRY POINT▸ Program Evaluators

can arrive at multiple points in program life–preferably early

Program Lifespan

Mature ImplementationEarlyPre

▸ Behavioral Scientists typically brought in late


DIFFERENCES: GRAIN SIZE

▸ Program • Behaviors▸ Program Components • Decision points


DESIGN THINKING▸ Involve behavioral

scientists at the start Program Lifespan

Mature ImplementationEarlyPre

▸ Summative evaluation remains purview of program evaluators

▸ Collaboration on formative evaluation


▸ When does a nudge become a program?

▸ When the nudge is a program, should we work from opposite ends?

CLOSING THOUGHTS


CLOSING THOUGHTS

How can we use reflective practice to learn from working together?


CONTACT INFORMATION

Anne Chamberlain443.259.5215

[email protected]

Documents

IMPAQ INTERNATIONAL LLC. PRESENTER: ANNE ......AEA 2016 – DESIGN THINKING: EVALUATION & BEHAVIORAL SCIENCE DESIGN THINKING Learning from the design process. Plan Do Study Act The