Upload
thomasina-chapman
View
219
Download
1
Tags:
Embed Size (px)
Citation preview
Building Support through
Effective Communication
Strategies
Building Support through Effective Communication Strategies
Robert KightChief, Division of WIA Adult Services and the Workforce System, ETA
Steven BakerVice President of Marketing and Communications, Jobs for the Future
Jacob KlermanPrincipal Associate/Scientist, Abt Associates
2
Here’s What to Expect
• Cover the basics of good story telling
• Framing your project’s story
• Develop a working draft
• Leveraging evaluation results
3
How to use your story
4
• Internal communications
• Talking points/speeches
• Presentations
• Funding proposals/reports
• Articles/publications
• Web/social media
• Press releases/media coverage
• Community engagement
Fundamentals of a good story
5
•It is clear and understandable, no matter how technical the work may be
•It is relatable; the story connects the work to people
•It is compelling; the outcomes/benefits are easily understood
•It is motivating; your audience wants you to succeed and they want to help you
Framing your story
6
First decide:• Who are your primary audiences?
• Program participants, businesses, funders, community leaders, policy makers
• What do they care about?• Populations, efficiencies, outcomes
• What will you want them to do?• Give direct support• Advocate on your behalf
Drafting your story
7
• What are the problem(s) you’re trying to solve? Frame in terms of both systems and people
• What is your solution? How will it help? Avoid jargon; keep it simple
• How is your solution unique? What’s innovative? Don’t forget to take credit; you’re the hero!
• What are the intended outcomes? What will be different? Better?
Who will benefit? Why should people care?
Using Evaluation Results
Jacob Alex Klerman
WIF Grantees Conference
Washington DC, March 2014
Abt Associates | pg 9
Outline
WIF as the “Vanguard”
Impact Results
Earlier Results
An Exercise
Abt Associates | pg 10
Outline
WIF as the “Vanguard”
Impact Results
Earlier Results
An Exercise
Abt Associates | pg 11
WIF Grantees are the “Vanguard”
Until recently, social policy in general and workforce strategies in particular have been set and funds allocated based on “plausibility”– And, rigorous impact evaluation suggests that impacts have often
been mediocre
Increasingly (Obama Administration’s Evidence Agenda, ED’s i3, CNCS’s SIF), we see a move towards a new and better strategy: “Evidence Based Policy”– Pilot, rigorously (impact) evaluate, replicate
– Broad scale rollout only after success at earlier steps
– Leading to better programs and (any and) larger impacts
WIF is a key component of the strategy
Abt Associates | pg 12
Perhaps “Bleeding Edge” … Being part of the “vanguard” is not easy
We are moving towards “evidence based policy” exactly because, when evaluated, many programs will not be found to be effective– So participating in WIF—and being evaluated—is the “right thing”
– But, the results are not always pleasant
The rest of this talk considers how to deal with that reality
Abt Associates | pg 13
Turn Burden into Selling Point In your materials, note that you are “doing the right thing”
– Stepping through the tiers of evidence
And, along the way, showing that:– Your program can actually implemented
(i.e., a successful pilot; see third part of talk)
– You are incremental tweaking your program(again, see third part of talk)
– You can work constructive with an evaluator
– You have good internal data systems that can support an evaluation
– And, we hope building increasingly high quality evidence of impact(see the next section of the talk)
… which are valued by “Evidence Based Policy” funders
Abt Associates | pg 14
Outline
WIF as the “Vanguard”
Impact Results
Earlier Results
An Exercise
Abt Associates | pg 15
Talking about Results;Especially if they are “Good”
Evaluator should produce a detailed report– Background, methods,
data, results
Someone (evaluator or grantee) should produce a brief “Executive Summary”– Standard format is one
page, two sides
– See handout with example
Abt Associates | pg 16
Talking about Results;Especially if they are “Good”
Evaluator should produce a detailed report– Background, methods,
data, results
Someone (evaluator or grantee) should produce a brief “Executive Summary”– Standard format is one
page, two sides
– See handout with example
Executive SummaryStandard Outline
Overview: One paragraph program and findings
Background Program Design Methods Implementation findings Impact findings, with 1-2
figures Discussion References
You really need to do this, whether results are “good” or “bad”
Abt Associates | pg 17
Strength of Methodology “Evidence Based Policy” is associated with “tiered evidence”
– Programs move up the “tiers of evidence”
– Pilot->pre/post->QED->random assignment->replication
– And only then to broad program roll-out
Claim credit for working through the tiers, but, don’t over-state the strength of the evidence– Random assignment provides the strongest evidence of impact
– Pre/post and QED tend to over-state impact
– They should, therefore, be taken (primarily) as evidence that moving to the next tier of evidence is appropriate
Exercise 1: Grantees and evaluators should discuss the evaluation’s methodology and how they will describe it and its strength
Abt Associates | pg 18
Consider Precision Carefully
A nearly ideal example: SEIS/Sectoral Employment Impact Study (Maguire, et al. 2010)– Random assignment evaluation found clear evidence of
larger impacts
– Impact on total earnings of 24 months since randomization
Why do we (“the policy community”) say that?
Abt Associates | pg 19
Consider Precision CarefullyCase 1/SEIS
Impact $4,509 This is a very large impact;18% of control
Standard Error $1,286 Relative to impact this is a small number
T-statistic 3.5 = Impact / Standard Error
P-value p<0.01 **** If no impact, this would happen less than 1% of the time
95% Confidence Interval $1,806 - $9,785 Range of plausible (95% of time)
impacts does not include zero
Language “Strong Evidence” When p<0.95
Abt Associates | pg 20
Several Very Different CasesCase 1/SEIS Case 2 Case 3 Case 4
Impact $4,509 $4,509 $2,000 $200
Standard Error $1,286 $2,652 $1,286 $129
T-statistic 3.51 1.70 1.56 1.56P-value p<0.01 p<0.10 p>0.10 p>0.10
95% Confidence Interval
$1,806 – $9,785
-$689 –$9,707
-$521 –$4,521
-$52 –$452
Language “Strong evidence”
“Limited evidence”
“No evidence”
“No evidence”
Abt Associates | pg 21
Several Very Different CasesCase 1/SEIS Case 2 Case 3 Case 4
Impact $4,509 $4,509 $2,000 $200
Standard Error $1,286 $2,652 $1,286 $129
T-statistic 3.51 1.70 1.56 1.56P-value p<0.01 p<0.10 p>0.10 p>0.10
95% Confidence Interval
$1,806 – $9,785
-$689 –$9,707
-$521 –$4,521
-$52 –$452
Language “Strong evidence”
“Limited evidence”
“No evidence”
“No evidence”
Just Can’tTell
No Impact
Very different cases; “Just Can’t Tell” is more common
Abt Associates | pg 22
Outline
WIF as the “Vanguard”
Impact Results
Earlier Results
An Exercise
Abt Associates | pg 23
Logic Model for Training Welders
Inputs OutputsOutcomes
Short-Term Long-Term
• Staff• Space• Materials• Partnerships
• Sessions held
• Teaching with fidelity
• Internships
• Attend sessions• Graduate• Pass external
exam• Employed in
target industry
• Earnings• Transfer
program participation
• Taxes paid
Abt Associates | pg 24
So Far, Discussed Impact
Inputs OutputsOutcomes
Short-Term Long-Term
• Staff• Space• Materials• Partnerships
• Sessions held
• Teaching with fidelity
• Interships
• Attend sessions• Graduate• Pass external
exam• Employed in
target industry
• Earnings• Transfer
program participation
• Taxes paid
“Impact”: Treatment vs. Control for “Long-Term Outcomes”– Not revealed until “later”
– Not useful for short-term adjustments
Abt Associates | pg 25
Use Logic Model to Refine Program
Inputs OutputsOutcomes
Short-Term Long-Term
• Staff• Space• Materials• Partnerships
• Sessions held
• Teaching with fidelity
• Internships
• Attend sessions• Graduate• Pass external
exam• Employed in
target industry
• Earnings• Transfer
program participation
• Taxes paid
Many “early steps” (inputs, activities/outputs, and short-term outcomes) can be observed early on, in your treatment group, at low cost
Abt Associates | pg 26
Learning from Early Results
A program’s Logic Model describes necessary—but perhaps not sufficient—early steps to achieve meaningful impact
You are currently implementing the program; you can check those early steps– Epstein and Klerman (2012) note that you will often find that the
early steps are not achieved
If your program is not achieving early steps, you can adjust the program design and implementation– Now, during this implementation (before impact results are known)
– For the next implementation (in response to impact results)
Abt Associates | pg 27
Some Early Steps to Check1. Secured partnerships?
Recruited and retained the right staff?
2. Attracted target number of participants?
3. Do Enrolled participants complete (enough of) the program?
4. Is the program implemented with fidelity?
5. Do participants show progress on pre/post measures of the program’s short-term outcomes?
6. Do participants pass external exams?
7. Do participants find employment in targeted industry?
Program details matter; examples have caveats
Abt Associates | pg 28
An Exercise
WIF as the “Vanguard”
Impact Results
Earlier Results
An Exercise
Abt Associates | pg 29
Exercise 2: Motivation
Epstein and Klerman (2012) argue that:– Even w/o an impact evaluation, we can establish that some
(many?) programs are unlikely to show impact
– Because they don’t “succeed” in the earlier steps of their own logic model
This exercise attempts to help you to be ready to use that insight constructively: – In this grant cycle, and in future grant cycles
Abt Associates | pg 30
Exercise 2: Part 1
Who: Grantee and evaluator, together
Materials: A copy of your program’s “Logic Model”– Ideally, a figure/graphic and the narrative discussing it
Part 1: Walk through your logic model, identifying each of the verifiable early steps– Inputs acquired
– Activities and outputs produced
– Outcomes achieved—in the treatment group, during or at the end of treatment
Abt Associates | pg 31
Exercise 2: Parts 2-5
Part 2: For each step, define “success”– Often defining success will require a quantitative standard
(e.g., number of trainees enrolled, percent of classes actually attended)
Part 3: Establish how and how early you can (and will) easily measure “success” for each earlier step
Part 4: If not “successful” at an early step of your logic model, discuss how you will “adjust” your program– During this grant period or in the next grant period
Step 5: If we identify common issues, we can try to arrange technical assistance
Abt Associates | pg 32
For More on these Ideas
Wholey, Joseph 1994. ‘‘Assessing the Feasibility and Likely Usefulness of Evaluation.’’ In Handbook of Practical Program Evaluation, edited by H. P. Hatry, J. S. Wholey, & K. E. Newcomer. San Francisco, CA: Jossey-Bass.
Epstein, Diana and J.A. Klerman. 2012. “When is a Program Ready for Rigorous Impact Evaluation?” Evaluation Review. 36(5): 373-399.
Abt Associates Policy Brief: When is a Social Program Ready for Rigorous Impact Evaluation?
Abt Associates | pg 33
Outline
WIF as the “Vanguard”
Impact Results
Earlier Results
An Exercise