Embed Size (px)
DESCRIPTION
The conversation around 21st century data collection methods continues to evolve, but little work has been done to employ these methods to empower community stakeholders as part of the data collection process. This session will detail how DCPNI conducted a representative community survey using innovative data collection methods within an inclusive evaluation framework by employing community members as agents of community data collection. We will address the challenges faced in implementing a survey of significant scope, including time constraints, unknown literacy levels of survey participants, and inconsistent access to the internet, and detail how these challenges were overcome. The session will include a step-by-step demonstration of DCPNI's use of Key Survey Software to administer surveys on tablet devices with offline capability. We will discuss the strengths and weaknesses of our approach compared to alternatives and outline how the collected data will be utilized to benefit the community going forward.
Citation preview
When Technology Hits the Sidewalk: Empowering community residents through 21st century data
collection tools
www.dcpni.org@dcpni
www.keysurvey.com@WorldAPP
American Evaluation Association Conference28th Annual Conference
October 16th 2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
2
Today’s Panelists
10/16/2014
Isaac CastilloDeputy Director
DC Promise Neighborhood Initiative
@isaac_outcomes
Samantha GreenbergData and Evaluation Specialist
DC Promise Neighborhood Initiative
@eval_revolution
Jim SouzaBusiness Analyst
WorldAPP, [email protected]
Billy MorrellDirector, Sales Engineering
WorldAPP, [email protected]
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
3
When Technology Hits the Sidewalk: Empowering community residents through 21st
century data collection tools
• The DC Promise Neighborhood Initiative’s Neighborhood Survey
• Including community residents in a neighborhood survey
• Data Collection in the 21st Century• Our successes and challenges
10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
4
The DC Promise Neighborhood Initiative’s Neighborhood Survey
10/16/2014
Isaac CastilloDeputy Director
DC Promise Neighborhood Initiative
Twitter: @Isaac_outcomes
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
5
As part of the White House Neighborhood Revitalization Initiative, Promise Neighborhoods align federal funding streams that invest in transforming neighborhoods of concentrated poverty into neighborhoods of opportunity.
10/16/2014
What is the Promise Neighborhood Approach?
The INSPIRATION….Harlem Children’s Zone (HCZ) inspired the U.S. Department of Education’s Promise Neighborhoods program, which launched in 2010.
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
6
What is DCPNI?
• DCPNI is one of 12 communities awarded funding from the U.S. Department of Education.
• DCPNI’s focus is on the Kenilworth-Parkside neighborhood of Washington, DC.
• Founded as an independent nonprofit in 2012 shortly after receiving federal funding.
10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
7
Kenilworth-Parkside by the Numbers
• 5,725 people – 1,800 children• 98% African American, 55% are female• 50% of adults live in poverty• 49% of K-P residents experience food
insecurity• 70% of K-P residents have a high school
education or higher (compared to 88% in DC)• 85% of households with children are headed by
single females10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
8
DCPNI’s Vision
To end intergenerational poverty in the
Kenilworth-Parkside community.
10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
9
CARING ADULTSin their lives as parents, mentors, teachers, coaches & resources
SAFE PLACESIn which to live, grow and learn
A HEALTHY STARTand healthy future
AN EFFECTIVE EDUCATIONthat equips children and parents with marketable skills
AN OPPORTUNITY TO GIVE BACK to one’s community
THE FIVE PROMISES
10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
10
How Does DCPNI Measure Progress on Each Promise?
• DCPNI has 15 federal indicators that we are required to track by the Department of Education.
• Developing additional indicators on issues that are unique to Kenilworth-Parkside. – Teen birth rate– Mother’s educational attainment– Food insecurity
• These indicators help us determine if we are progressing towards the Promises.
10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
11
Sample of DCPNI Indicators
• Parents read to their 0-5 year old children.
• Increased feelings of safety among community residents.
• Children ages 0-5 that have a medical home.
10/16/2014
These data points need to be collected directly from community residents.
SAFE PLACES
HEALTHY START
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
12
The Need for a Neighborhood Survey
• Collect information directly from community residents.
• Limited other publically (or privately) available data sets for the Kenilworth-Parkside population.
• Desire to measure community level change over time.
• What little data that was available was old, limited to parts of the community, or collected using low rigor methods.
10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
13
The Most Significant Challenges
• Very short timeframe – needed to complete entire process (from survey design to final analysis of data) in 9 months.
• Low levels of literacy in the community (and low comfort level with technology).
• High levels of suspicion of outsiders (and even suspicion of those from another part of the community).
• High levels of distrust of government and government-like entities.
10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
14
Including Community Residents in a Neighborhood Survey
10/16/2014
Samantha GreenbergData and Evaluation Specialist
DC Promise Neighborhood Initiative
Twitter: @eval_revolution
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
1510/16/2014
Be inclusive
2. Gain community support for survey
Build relationships
Be culturally responsive
Add unique value
Goals for Kenilworth-Parkside Survey:
1. Use technology to overcome common survey challenges
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
1610/16/2014
Common Survey Concerns:1. Cost
2. Confidentiality
3. Standardization
4. Flexibility
5. Ease
Maximize inclusivity and overcome survey challenges through technology
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
17
• Used KeySurvey as our survey platform
Allowed ability to build conditional sections and complicated skip patternsAllowed personalization based on household traits• Cost-reducing in terms of time, money and flexibility
10/16/2014
Program
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
1810/16/2014
Feature Paper Surveys Tablet Surveys
MoneyPrinting supplies,
associated materials, postsurvey dataentry
$200 per tablet, KeySurveysubscription
ConfidentialityMaintaining security of
documents in field, postsurvey
Tablet’s secure connection and privacy capabilities
Standardization Interview cues
Flexibility May involve skipping pages Skip logic, conditionality
Ease
Paper vs. Plastic
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
19
Survey Administration Teams:
10/16/2014
Kenilworth-Parkside
Community Resident
• Navigated the neighborhood• Greeted person at door• Introduced survey• Documented households visited • Handled incentives
• Consent process• Read survey questions aloud• Documented answers on tablet• Handled technology
Gain community support for survey
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
2010/16/2014
Community residents:• Familiar with neighborhood• Have relationships with survey respondents• Knowledgeable about community history and assets
Step 1: Build an Inclusive Process
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
2110/16/2014
City Year corps team members :• Versed in technology• Able to offer confidentiality (not from K-P)• Experienced serving D.C. communities
Step 1: Build an Inclusive Process
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
2210/16/2014
Advantages of Two-person Teams
• Increased likelihood of “open doors”• Decreased # of survey days needed• Easy navigation of community• Division of labor• Two people available to answer questions• Balance between confidentiality and trust• Increased safety
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
2310/16/2014
Step 2: Prioritize cultural sensitivity
• Survey questions tested with community
• Community feedback encouraged throughout
• Survey administrators participated in focus group
• Concerns from community members noted for next survey
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
2410/16/2014
Step 3: Add Unique Value
• I heart data booklet provides data snapshot
• Data now exists on topics for which there are no other sources
• Ongoing conversations with the K-P community about data
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
2510/16/2014
Step 4: Build Relationships
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
2610/16/2014
Inclusive data collection
2. Gained community support for survey
Built relationships
Culturally sensitive survey
Add unique value
Goals for Kenilworth-Parkside Survey:
1. Addressed common survey challenges with technology
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
27
The Use of WorldAPP Offline Technology for a Neighborhood Survey
10/16/2014
Billy MorrellDirector, Sales Engineering
WorldAPP, [email protected]
Jim SouzaBusiness Analyst
WorldAPP, [email protected]
Data Collection in the 21st Century
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
2810/16/2014
The WorldAPP Story
WorldAPP is the global leader in data collection and business process optimization software. Whether it’s streamlining data capture, automating form-driven operations, or dramatically improving data analysis, WorldAPP delivers unique cloud-based solutions configured to your specific enterprise needs. Since beginning operations in 2002, WorldAPP has become the premier solutions provider and a strategic technology partner for many of the world's largest enterprises.
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
2910/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
30
• Level one: 159 questions and hundreds of answer choices
• Level two: Variable names
10/16/2014
DCPNI Survey: Back-End vs. Front-End
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
31
• Level Three: Skip patterns– Sometimes a simple skip:
10/16/2014
DCPNI Survey: Back-End vs. Front-End
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
32
• Sometimes it’s something else entirely:
10/16/2014
DCPNI Survey: Back-End vs. Front-End
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
33
• Autofill of information into further questions:
10/16/2014
DCPNI Survey: Back-End vs. Front-End
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
34
Out of the Box Reporting
10/16/2014
DCPNI Survey: How it could have worked even better
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
35
• Online Data Collection• Automatic Email Alerts and Workflows• Calculations• Integrations
10/16/2014
Additional Applications
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
36
Our Challenges
• Sometimes technology fails – 4 tablet failures in the middle of survey administration.
• Typing answers in leads to some data quality errors (misplaced or missing variables)
• Tablets and technology require investments of time and training:• Survey must be built into WorldApp and then tested on
tablets (time-intensive programming)• Administrators must be trained in use of tablets and how
to problem solve.
10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
37
Our Successes
• Data ready for cleaning 24 hours after last survey was administered.
• Data ready for analysis three days after last survey was administered.
• Data security high (end of day wipe, Cerberus for tablet security)
• 444 completed surveys over 10 days (response rate of 59%).• Community buy-in for survey and highly attended release
of findings during I Heart Data event.
10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
38
Audience Q&A
10/16/2014
Isaac CastilloDeputy Director
DC Promise Neighborhood Initiative
@isaac_outcomes
Samantha GreenbergData and Evaluation Specialist
DC Promise Neighborhood Initiative
@eval_revolution
Billy MorrellDirector, Sales Engineering
WorldAPP, [email protected]
Jim SouzaBusiness Analyst
WorldAPP, [email protected]
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
39
It Takes a Neighborhood to Design a Survey: The Methodology Behind the Kenilworth-
Parkside Neighborhood Survey
10/16/2014
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
40
Methodology Overview
10/16/2014
• Instrument Design• IRB Review• Sampling• Sampling frame• Strata, or key groupings• Random selection
• Response Rates • Population Weights
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
41
Instrument Design
10/16/2014
• Create a baseline assessment of the neighborhood from which to assess change
• Many considerations, including • Department of Education Government Performance and
Results Act (GPRA) indicators• DCPNI programmatic priorities• DCHA Choice Neighborhoods data needs• Urban Institute suggestions• Resident interests and concerns• Instrument length, nature of questions
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
42
Instrument Design
10/16/2014
• Topics include:• Housing• Neighborhood amenities• Access to Food • Neighborhood Supports • Neighborhood Conditions and Safety• Household Composition• Travel, Education, Employment and Public Assistance• Child Health, Education, and Well-being
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
43
IRB Review
10/16/2014
• Institutional Review Board (IRB) makes sure that research minimizes risks to human subjects
• Reviewed and approved the plan for:• Requiring confidentiality pledges from staff• Obtaining consent from research subject• Roles for community resident and City Year corps
member• Providing incentives for completed surveys• Keeping data confidential
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
44
Sampling
10/16/2014
• Used a list of addresses for 2 census tracts that make up Kenilworth Parkside
• Separated them into 3 sub-neighborhoods, or strata• Group 1: Kenilworth Courts• Group 2: KPRMC, Mayfair, Paradise, Lotus Square and
Victory Square • Group 3: Eastland Gardens and Paradise
• Randomly select addresses within each sub-neighborhood (SAS Proc Surveyselect)
• Oversampled Group 1
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
45
Response Rate
10/16/2014
• Eligible households• 872 addresses randomly selected• 115 unoccupied• 872 – 115 = 757 eligible households
• Completed surveys• 444 completes
• Response rate• 444 / 757 = 59% response rate
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
46
Population Weights
10/16/2014
Sub-neighborhood Population
% of Population
Completed Surveys
% of Surveys Weight
Group 1: Kenilworth Courts
283 12% 133 30% =12%/30%=.3857
Group 2: KPRMC, Mayfair, Paradise, Lotus Square and Victory Square
1620 68% 217 49% =68%/49%=1.4012
Group 3: Eastland Gardens and Paradise
476 20% 94 21% =20%/21%=.9731
Total 2379 100% 444 100% 1.0000
D.C. Promise Neighborhood Initiative and WorldAPP, Inc.
4710/16/2014
Back-End v. Front-End
