Developing a Data and Evaluation Framework for Urban Promise Zones · 2019. 3. 15. · Explain how data can help organizations build their own capacity ... Developing a Data and Evaluation

May 2018

Developing a Data and Evaluation Framework for Urban Promise Zones

Developing a Data and Evaluation Framework for Urban Promise Zones: Session 1

Using WebEx Chat and Q&A Features

2

Chat Feature (for help with technical Issues)• Select “Host” in “Send To” box• Type message• Click “Send”

Q&A Feature (to send a question to the presenters)• Select “All Panelists” in “Send To” box• Type message• Click “Send”

May 2018

Developing a Data and Evaluation Framework for Urban Promise Zones


Poll Quiz White Board

Exercise Sharing Q&A

Classroom Involvement

Discussion


Introduction, Overview, & Roles

POLL: What is your role in the Promise

Zones initiative?


Learning Outcome and Objectives of the Series Learning outcome

• Successfully develop a data and evaluation framework that tracks and evaluates the PZ’s progress toward achieving its goals and thus improving programmatic effectiveness.

Learning objectives• State the importance of collecting data and identify common measures for place-based

initiatives • Define the elements of an evaluation framework for place-based initiatives and be able to

apply that knowledge • Strategize how to leverage partner and other resources to gather both qualitative and

quantitative data to measure progress in achievement the PZ’s goals• Effectively analyze data and create effective data visualization to support data-driven policy

7

POLL: Which statement best characterizes your PZ’s current

capacity to develop and implement a data and evaluation

framework?


Introduction and overview Review data and evaluation glossary purpose

Discuss roles for PZs, partner organizations, and Vistas

Describe Theory of Change (TOC) and logic models and discuss their relevance for the data and evaluation framework

9

Define “data and evaluation framework” Discuss how data and evaluation framework and

TOC can support PZ’s performance Describe elements in a data and evaluation

framework Compare types of evaluations and their purposes

Describe data sources and measures with Group A, B, C, D indicators

Discuss types of data Describe systems and practices for collecting and

storing data, including data usage agreements Share common challenges and solutions in data

collection

Discuss best practices for data analysis, including data visualization

Demonstrate data visualization tools and resources Analyze how data are used to drive implementation

and decision making Discuss techniques for reporting and dissemination

to stakeholders

Series OverviewSe

ssio

n 1

Sess

ion

2

Sess

ion

3Se

ssio

n 4


Additional Supports Office Hours: One-on-one help to assist PZs as they complete

the homework and work to complete or update their data and evaluation frameworks Peer-to-Peer Learning: 4 sessions designed to connect PZ staff

for mutual learning and support

10


Manolya Tanyu

Your TA Provider Team

11

Xiaodong Zhang

Webinar Trainers (ICF)

Office Hours and Peer to Peer Sessions

(AIR)

Kasia RazynskaJanet Pershing

Trish Campie Jennifer Loeffler-Cobia


Participant’s Role Stay engaged, don’t multi-task Share your experience Ask questions! Do the homework Take advantage of office hours

12

QUIZ: Data visualization is a term referring to a very specific set of

displays that can be used to analyze data that require a specialized

software to generate. (T/F)

QUIZ: The important distinction between primary and secondary data

is timing: primary data are data collected prior to the start of an

evaluation while secondary data are data collected after the evaluation.

(T/F)

WHITE BOARD:CONFUSING GLOSSARY TERMS?


Place-Based Initiatives HUD/USDA: Promise Zones HUD: Strong Cities Strong

Communities (SC2) HUD: Choice Neighborhoods ED: Promise Neighborhoods SAMHSA/ED/DOJ: Prevention

Prepared Communities

14


What Is Collective Impact? Organizations from different sectors agree to collaborate to solve a specific social

problem or achieve shared outcomes. Elements needed:

• Influential champion• Adequate financial resources• Sense of urgency for change

Three phases of Collective Impact Process• Initiate action• Organize for impact• Sustain action and impact

17

From Channeling Change: Making Collective Impact Work, Stanford Social Innovation Review, Fay Hanleybrown, John Kania & Mark Kramer, January 2012. For information go to: http://www.ssireview.org/blog/engry/channeling_change_making_collective_impact_work


Effectively Engaging Partner Organizations Engage partner organizations early on to help define overarching

goals and accompanying data and evaluation framework Explain how data can help organizations build their own capacity

and achieve organizational goals Clearly outline data collection expectations from the start Provide resources and technical assistance to collect requested

data

18

SHARING: ENGAGING PARTNERS AND

STAKEHOLDERSKayleigh Creswell, San Diego Promise Zone


Theory of Change and Logic ModelThis section of the webinar is adapted from Long, M., and MacDonald, A. (2017). Practical Tips for Developing and Using

Theories of Change and Logic Models. Prepared for Volunteer for West Virginia. ICF.

POLL: How comfortable are you with your understanding of Theory of Change

and Logic Model?


What Is a Theory of Change (TOC)?

Presents a clearly expressed relationship between the three core elements:• Populations (Who)• Outcomes (What)• Strategies (How)

22

This is the foundation of your data and evaluation framework

Describes how and why desired changes are expected to happen in a particular context.


Why TOC? Discussion of the TOC helps partners reach consensus by

articulating: Assumptions such as how the program will work and how it will

help address the problem Goals the program is trying to achieve, and for whom Contribution that the program makes to achieve these goals

23


PIT-B Approach to Theory Of Change Development

24

Problem If

Then Because


Adolescents in our city suffer from high levels of obesity and related ailments. We believe adolescents do not know how to make good food choices, and often do not understand the impacts that those choices can have on their health. Our theory is that if adolescents learn about healthy shopping and cooking, then those students will become healthier. This is because they will learn about how to choose better food options and develop positive attitudes toward eating. Our program helps students learn about choosing and cooking healthy meals by delivering a fun and interactive after-school program. By providing adolescents with an age-appropriate nutrition curriculum, we will improve their knowledge and attitudes about healthy food. Ultimately, this will reduce adolescent obesity and related ailments.

Problem If

Then Because

PIT-B EXERCISE


Adolescents in our city suffer from high levels of obesity and related ailments. We believe adolescents do not know how to make good food choices, and often do not understand the impacts that those choices can have on their health. Our theory is that if adolescents learn about healthy shopping and cooking, then those students will become healthier because they will learn about how to choose better food options and develop positive attitudes toward eating. Our program helps students learn about choosing and cooking healthy meals by delivering a fun and interactive after-school program. By providing adolescents with an age-appropriate nutrition curriculum, we will improve their knowledge and attitudes about healthy food. Ultimately, this will reduce adolescent obesity and related ailments.

Problem If

Then Because

PIT-BANSWER KEY


Common Pitfalls To Avoid• Circular Logic: Repeating the same statement in different words• Process Focused: Going into the weeds on the “how” of your program

but forgetting to address the “why”• Program Histories: Giving a detailed looked at every way your program

has evolved, but forgetting to explain why it works the way it does• Literature Reviews: Describe the results of research into similar

programs, but do not address the underlying factor behind those programs

27

Q & A

DISCUSSION: How can you use a TOC to support

a data collection and evaluation framework?


Use Your TOC!

30

Do not stop with simply articulating your TOC It is a tool to guide your PZ work! In coming webinars we will discuss how to:

• Use your TOC as a guide to articulating interim activities and outcomes will result in the long-term change you seek.

• Target your data collection to those key elements• Use the data to make evidence-based decisions about ways to

improve your program.


Connection Between a TOC and a Logic Model

28

A TOC is a narrative description of why your program works A logic model translates this why into a fleshed out how

A logic model is a visual representation of the way your program will accomplish the changes established in your TOC


Why Logic Model? Reflects the program’s complexity and teases apart the distinct, but

inter-related, elements. Forces a clear, structured articulation of the program, which helps

all partners ensure they are on the same page. Helps uncover whether parts of the program do not fit or need re-

working

29

Developing a Data and Evaluation Framework for Urban Promise Zones: Session 1 33

Inputs Activities OutputsOutcomes

Short Medium Long

Key Components of a Logic Model: Inputs The human, financial, organizational, and community resources available

for carrying out a program’s activities: • Funding • Program staff • Volunteers

Outcomes • Research

Inputs Activities Outputs Short Medium Long


Key Components of a Logic Model: Activities Processes, tools, events, and actions that are used to bring about a

program’s intended changes or results: • Workshops on healthy food options • Food preparation counseling • Referrals to food programs and resources

Outcomes



Key Components of a Logic Model: Outputs Direct products of a program’s activities and may include types, levels and

targets of services to be delivered by the program: • # individuals attending workshops • # individuals receiving services • # individuals receiving referrals

Outcomes



Key Components of a Logic Model: Outcomes Expected changes in the population served that result from a program’s activities and fall along a

continuum, ranging from short to long term results: • Short-term (1-3 yrs): changes in knowledge, skills, and/or attitudes (e.g., ↑ knowledge healthy

choices) • Medium-term (3-5 yrs): changes in behavior or action (e.g., ↑ adoption of healthy food practices) • Long-term (5-10 yrs): changes in condition or status in life (e.g., ↑ improved health)

Outcomes



Comparison Between Outputs and Outcomes

Outputs Outcomes

• Direct count of a program’s activities/services

• Changes resulting from a program’s activities/services

• Examples: • # attending workshops • # receiving services • # receiving referrals

• Examples: • ↑ knowledge of healthy choices • ↑ adoption of healthy practices • ↑ improved health


Outcomes Inputs Activities Outputs

Short-term Medium Long-term

What resources go into a program

What activiti es the program

undertakes

What i s produced

through those activities

Changes or benefits that

result from program in near-term


occur in l onger timeframe

Long-term changes or benefits of

program, often at soci al level

e.g. money, staff, curriculum

e.g. deli ver training

programs

e.g. number of workshops held, people trained

e.g. improved knowledge/skills

/attitudes

e.g. improved food choices

e.g. improved health in

community

Putting it Together


Putting It Together

Inputs Activities Outputs Outcomes

Short-term Intermediate Long-term Changes or benefits that result from program in near-term


program, often at social level

What is produced



occur in longer timeframe


What activities the program undertakes

AB


C A B

Putting It Together

Inputs Activities Outputs

What is produced



What activities the program undertakes

Outcomes

Short-term Intermediate Long-term Changes or benefits that result from program in near-term


occur in longer timeframe


program, often at social level



Short Long

Funding/Budget

Human Capital: Program

Staff/Community Stakeholders

Collaboration in program Services

and Supports

Research about best practices

Demolition, construction, rehabilitation, preservation. Com Building

Community improvements

(e.g., public transit, banks,

schools, services, parks); public

safety

Number of units at diff. rents; demolished, rehabilitated,

preserved – Mix of units

Number of community

improvements; less crime

Distressed properties are

revitalized

Improved, quality community

services

Increased trust between

stakeholders, improved

community engagement

Medium

High quality, energy-efficient,

affordable housing units in mixed-income developments

Safe, healthy,mixed-income neighborhoods

with high-performing

schools and services

Transformed Neighborhoods



Short Medium Long

If Because Then

Demolition, construction, rehabilitation, preservation. Com Building

Community improvements

(e.g., public transit, banks,

schools, services, parks); public

safety

Number of units at diff. rents; demolished, rehabilitated,

preserved – Mix of units

Number of community

improvements; less crime

Distressed properties are

revitalized

Improved, quality community

services

Increased trust between

stakeholders, improved

community engagement

High quality, energy-efficient,

affordable housing units in mixed-income developments

Safe, healthy, mixed-income neighborhoods

with high-performing

schools and services

Transformed Neighborhoods

Problem If

Then Because


Developing a Data and Evaluation Framework for Urban Promise Zones: Session 1 44http://ohcdphila.org/wp-content/uploads/2015/11/neighborhood-plan-pages-35-to-49.pdf

Tips for Creating Logic Models It may help to borrow elements of a logic model from another organization

with a similar mission. Be sure to adapt it so it reflects your specific program. Look around the internet for examples Think about how to translate your PIT-B Problem If Then Because

statement to a logic model In a place-based setting, you may start with sector-based logic models, then

combine to a PZ-level logic model. A logic model is a living document—don’t be afraid to adjust it as you learn

through implementation


Q & A

SHARING: TOC/LOGIC MODEL CHALLENGES

Kayleigh Creswell, San Diego Promise Zone

Homework Building blocks of the Data and Evaluation Framework Format options • Use template provided • Use your own template Support Available! Sign up for Office Hours


1. Working with your team, develop or revise the Theory of Change and Logic Model for your PZ to clearly define inputs, activities, outputs, and outcomes (short-, medium-, and long-term) as well as connections among these elements.

Homework

Build or Refine Your Logic Model:


Homework 2. List the barriers that may make it hard to implement your Logic Model and solutions you intend to pursue.


Homework documents are available at


Homework https://hudexchange.info/programs/promise-zones/data-and-evaluation-framework/ Homework Instructions Logic Model Template Barriers Analysis Template

51

https://hudexchange.info/programs/promise-zones/data-and-evaluation-framework/


Resources HUD Exchange PZ Webinar Page:

https://hudexchange.info/programs/promise-zones/data-and-evaluation-framework/ Office Hours on Tuesdays, Wednesdays, and Thursday, the week

following each webinar: Sign up at: http://pzofficehours.simplybook.me Peer to Peer Learning: (3-4 PM EDT)

May 9 May 23 June 14 July 9 Webinars: (1-3 PM EDT)

May 16 June 7 June 27 52

https://stage.hudexchange.info/programs/promise-zones/data-and-evaluation-framework/

http://r.postserver.simplybook.me/tr/cl/P255IBTBf2tWIjjsbPsRK2k2RxIHzQpFNp41SXASM5Aga9JQRgLzZyR6Xa8uA9ItmrDZwCNu4aypJ5hyYUvYe30Euh16zbb4O1agL9xu0zfQwtPPu7OX2Gfc3YUeEnp0ejgLK5jJXpGLEfa2jIXyT5ATDFj9PYmg-iAkvh4sTs97C4uqPfSnX_IMV0OETZmvQtV-Fw


References CDC http://www.cdc.gov/evaL/resources/index.htm CNCS https://www.nationalservice.gov/resources/evaluation/all-

evaluation-resources W.K. Kellogg Foundation Logic Model Development Guide

http://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide Philadelphia Division of Housing and Community Development

Neighborhood Plan (pp. 46-49) http://ohcdphila.org/wp-content/uploads/2015/11/neighborhood-plan-pages-35-to-49.pdf

53

http://www.cdc.gov/evaL/resources/index.htm

https://www.nationalservice.gov/resources/evaluation/all-evaluation-resources

http://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide

http://ohcdphila.org/wp-content/uploads/2015/11/neighborhood-plan-pages-35-to-49.pdf

See You Again Soon! Office Hours: May 8, 9, and 10 from 1-5 pm EDTPeer to Peer Learning: May 9, 2018, 3-4 pm EDT

Webinar #2: May 16, 2018, 1-3 pm EDT Developing a Data and Evaluation Framework for Urban Promise Zones: Session 1

Documents

Developing a Data and Evaluation Framework for Urban Promise Zones · 2019. 3. 15. · Explain how data can help organizations build their own capacity ... Developing a Data and Evaluation