Copyright © 2018 by OneJustice. All rights reserved.
Qualitative & Survey Methods for Research & Evaluation
March 20, 2019
Copyright © 2018 by OneJustice. All rights reserved.
Orientation1. Intro to research & evaluation2. Designing projects + developing capacity3. Qualitative & survey methods4. Quantitative methods5. Synthesis, comms + decision-making
Copyright © 2018 by OneJustice. All rights reserved.
Course update1. Feedback: feeling more concrete; keep
focusing on exercises + practicalities2. Capstone update: projects are coming
together, nice work!3. Looking ahead:Ø Quantitative methods (04/03)Ø Capstone calls (04/11 – 04/12)Ø Capstone development periodØ Final session, inc. presentation (06/12)
Copyright © 2018 by OneJustice. All rights reserved.
Agenda for todayTime Topic
10:15am – 11:00amOverview of qualitative methods<<<Exercise: developing an interview guide>>>
11:00am – 11:15am Break
11:15am – 12:15pm Guest presentation: Bonnie Hough & Kim Tyda
12:15pm – 1:00pm Lunch
1:00pm – 2:15pm Overview of survey methods
2:15pm – 2:30pm Break
2:30pm – 3:45pm <<<Exercise: survey design workshop>>>
3:45pm – 4:00pm Wrapping up
Copyright © 2018 by OneJustice. All rights reserved.
Qualitative Methods
Copyright © 2018 by OneJustice. All rights reserved.
Listening Exercisehttps://www.legalaidsmc.org/videos/
From the client’s perspective, what is most important about the help they received from legal aid?
Copyright © 2018 by OneJustice. All rights reserved.
Why use qualitative methods?For capturing the human dimensionFor characterizing and understandingFor centering client voices
Opinions
Beliefs
Feelings
Reasons
Decisions
Experiences
ReflectionsPreferences
Copyright © 2018 by OneJustice. All rights reserved.
Key features of qualitative methods
1. Semi-structured2. Controlled exploration3. Active listening
Copyright © 2018 by OneJustice. All rights reserved.
Example: exploring self-representation Justice DivertedExperiences of tenants in rent court
Justice Diverted report:http://legalaidresearch.org/pub/4682/justice-diverted-how-renters-are-processed-in-the-baltimore-city-rent-court/
“The only question they ask you is, “So you agree with the amount that’s owed?” Like they don’t, you know, take the time to get your
side of the story or to see….why you haven’t paid anything, or if there’s anything wrong with the property.”
Copyright © 2018 by OneJustice. All rights reserved.
Example: value of representationVera Evaluation Experiences of NYIFUP clients
Vera Evaluation of New York Immigrant Family Unity Project:
https://storage.googleapis.com/vera-web-assets/downloads/Publications/new-
york-immigrant-family-unity-project-evaluation/legacy_downloads/new-york-
immigrant-family-unity-project-evaluation.pdf
“The legal system is a different kind of comprehension. They use
different kinds of words with legal definitions. So a regular word
would not be a regular word when applied legally…. If I wasn’t
provided a lawyer, I couldn’t stand a chance. I didn’t know the
law. Everybody in court needs a lawyer. ”
Copyright © 2018 by OneJustice. All rights reserved.
Capstone applications
Understanding client
experiences
Exploring staff and stakeholder
perceptions
Understanding how services
work
Developing + testing surveys
Copyright © 2018 by OneJustice. All rights reserved.
Planning qualitative projects1. Defining objectives2. Designing sample3. Selecting and tailoring a methodology4. Interview guide5. Fieldwork6. Analysis
Copyright © 2018 by OneJustice. All rights reserved.
Defining objectives
What topic(s) do you want to explore? (e.g.
benefits of representation to clients)
Try to identify the different strands of this
topic, e.g.
• Perceptions – lawyer’s role
• Experiences – relationship w/ attorney
• Attitudes – reflection on experience
Copyright © 2018 by OneJustice. All rights reserved.
Designing a sample• Goal is to capture the range or variety• Aim for a mix of known characteristics• Common sample sizes in range 10-30• Sample until responses are less varied• Create sub-samples (e.g. clients, judges)• Some variables only known at recruitment
stage (e.g. satisfied vs dissatisfied)• Think critically and consider biases
Copyright © 2018 by OneJustice. All rights reserved.
Selecting a qualitative methodology
Individual Group
ü Group dynamicü Efficient
ü Individual focusü Flexible
Copyright © 2018 by OneJustice. All rights reserved.
Tailoring a qualitative methodology
What is the best time and situation to
gather the feedback you want?
Credit: Margaret Hagan
https://medium.com/legal-design-and-innovation/a-service-design-approach-to-the-legal-system-ef01ae0c2b78
Copyright © 2018 by OneJustice. All rights reserved.
Developing an interview guideSemi-structured exploration of topicAiming for a intuitive flowAddress topic in multiple ways
Intro: welcome, consent, confidentiality
Personal & family background
Experience of contact with legal aid
Perceived strengths & weaknesses
Outro: final reflections & thanks
Copyright © 2018 by OneJustice. All rights reserved.
Questions & exercises
• Go for simple, open-ended, value-
neutral questions
• Don’t be afraid to probe (“Can you tell
me a bit more about that…?”)
• Use practical exercises to break up the
interview (e.g. journey mapping)
“How did you decide to contact Legal Aid for help?”
Copyright © 2018 by OneJustice. All rights reserved.
Recruitment & fieldwork
Early communicationRecruitment processIncentives
LocationPrivacyAccommodations
Copyright © 2018 by OneJustice. All rights reserved.
Interviews and data collection
Interviewers
• Staff or volunteers
• Language & cultural competence
• Consider trauma/traumatization
• Briefing, training & oversight
Options for data collection
Copyright © 2018 by OneJustice. All rights reserved.
Content analysis
Participant Strengths of service Weaknesses of service
1 Attorney good listener Irregular case updates
2 Attorney’s expertise Irregular case updates
3 Free representation Attorney poor listener
Copyright © 2018 by OneJustice. All rights reserved.
Presenting findings
Case studies & vignettes
Summarizing key themes
Copyright © 2018 by OneJustice. All rights reserved.
Interview Guide Exercise
Copyright © 2018 by OneJustice. All rights reserved.
Interview Guide Exercise
Copyright © 2018 by OneJustice. All rights reserved.
Time for lunch….
Copyright © 2018 by OneJustice. All rights reserved.
Survey Methods
Copyright © 2018 by OneJustice. All rights reserved.
Why use survey methods?For gathering info or feedback from peopleFor measuring, comparing & tracking
Strongly agree
Strongly disagree
“The attorney listened carefully to what I was saying.”
Copyright © 2018 by OneJustice. All rights reserved.
Key features of survey methods
1. Using samples to understand population2. Structured topic coverage via a
questionnaire (e.g. listening behavior) 3. Measurement of characteristics (e.g. %
agree attorney listens carefully)
Copyright © 2018 by OneJustice. All rights reserved.
Example: Justice Gap study
LSC 2017 Justice Gap Study: https://www.lsc.gov/media-center/publications/2017-justice-gap-reportCalifornia’s forthcoming Justice Gap study: http://www.calbar.ca.gov/Portals/0/documents/accessJustice/Justice-Gap-Study-fact-sheet-final.pdf
71% of low-income Americans experienced at least 1 civil legal problem in last year
Copyright © 2018 by OneJustice. All rights reserved.
Example: tracking outcomes
Cleveland Legal Aid’s Texting Project: https://medium.com/innovations-in-legal-aid/cleveland-legal-aid-learned-the-outcomes-of-its-brief-services-through-texting-4c24202d7a7b
“You recently attended a legal clinic for help filing a divorce. Did you file your papers with the court? Text
Y or N to let us know.”
Copyright © 2018 by OneJustice. All rights reserved.
Capstone applications
Client intake surveys and
baseline measures
Client exist surveys and
follow-up measures
Assessing learning and
behavior change
Surveying program
characteristics
Copyright © 2018 by OneJustice. All rights reserved.
Planning survey projects• Defining objectives• Sampling• Tailoring methodology• Questionnaire• Fieldwork• Analysis
Copyright © 2018 by OneJustice. All rights reserved.
Defining objectivesWhat question(s) do you want to answer? What information do you need?
Try to distinguish between priority and contextual and supplemental information
Copyright © 2018 by OneJustice. All rights reserved.
Sampling terminology
• Target population: a group of interest; often also defined by time and place
• Census: full population measured• Sample: a subset of population used to
make inferences about the whole • Sample stratification: splitting sample
into subgroups (e.g. by case type)
Copyright © 2018 by OneJustice. All rights reserved.
Sample size• Consider aiming for n = 50 as a minimum• Sample can be acquired over time (e.g.
quarterly data collection)• Margin of error is a function of variability
in sample and population size• Increase sample size if you want to• Reduce margin of error • Enable more robust comparisons
• Try using sample size calculators: e.g. https://www.surveysystem.com/sscalc.htm
Copyright © 2018 by OneJustice. All rights reserved.
Sampling procedure• Prepare your contact list (if necessary,
separate into ‘strata’)• Randomly select a large group to
attempt to contact (only some respond)• Work through sample in batches;
attempt each contact at least 2-3 times• Record status of (a) contact and (b)
participation and (c) completion• Monitor response rate and patterns
Copyright © 2018 by OneJustice. All rights reserved.
Sampling exampleTime period Clients opening
new casesContact list(10% of clients)
Responded (assuming 25% RR)
Q1 1,000 100 25
Q2 1,000 100 25
Q3 1,000 100 25
Q4 1,000 100 25
Total 4,000 1,000 100
A sample of 100 for population of 4,000
gives +/- 10% margin of error
Copyright © 2018 by OneJustice. All rights reserved.
Managing contact listsName Intake
DateLegal area Home Phone Mobile
phoneEmail Consent
Client A 12/01/19 Consumer 000-000-0000 000-000-0000 [email protected] Yes
Client B 12/01/19 Housing 000-000-0000 [email protected]
Client C 12/02/19 Housing [email protected] No
Client D 12/03/19 Housing 000-000-0000 000-000-0000 [email protected] Yes
Client D 12/03/19 Housing 000-000-0000 000-000-0000 [email protected] Yes
ü Deduplicated ü Contact info providedü Consent given
Copyright © 2018 by OneJustice. All rights reserved.
Are my results reliable?• All surveys contain error; goal is to avoid
erroneous conclusions• Best practice is to try to estimate error• Sampling errors (sample size)• Other biases (response bias)
• Importance depends on how data is used, esp. for decision-making
• Avoid assuming too much precision when sample size is small
Copyright © 2018 by OneJustice. All rights reserved.
Survey modes
Copyright © 2018 by OneJustice. All rights reserved.
Structuring a questionnaire
Introduction
Contextual info (if needed)
*** Priority section ***
Supplemental info (if needed)
Conclusion + exiit
Copyright © 2018 by OneJustice. All rights reserved.
Crafting questions
Copyright © 2018 by OneJustice. All rights reserved.
Binary QuestionsUse yes/no questions when:• It is an inherently binary situation• You want a very simple measure
Have you completed a training course on this subject before?
Did you file your papers with the court? Text Y or N to let us know.
Copyright © 2018 by OneJustice. All rights reserved.
Multiple select questionsUse when you want respondents to select all that apply (not mutually exclusive)
What kind of legal help did you receive? Check all that apply.
1. I was referred to legal information online2. I got assistance filling out a legal documents or forms3. I got legal advice4. A legal professional helped me negotiate with the
other person [...]5. I was represented by a legal professional in court
Copyright © 2018 by OneJustice. All rights reserved.
Multiple choice (unordered)Use when you want a response in only one of a mutually exclusive set of categories
Which of these statements best describes the status of your legal issue?
1. The issue is ongoing, and I’m still dealing with it
2. The issue is not resolved, but I haven’t been dealing with it lately
3. The issue is resolved
Copyright © 2018 by OneJustice. All rights reserved.
Multiple choice (ordered)Use these when you want a response that falls on a scale or range (unipolar or bipolar)
How much did this legal issue personally affect you?
1. Not at all2. Slightly 3. Moderately4. Very Much5. Severely
How strong is your peer support network?
1. Very weak2. Weak3. Fair4. Strong5. Very strong
Copyright © 2018 by OneJustice. All rights reserved.
Likert scalesUse these when you want to gauge agreement with a number of statements
“The attorney listed carefully to what I was saying.”
1. Strongly disagree2. Disagree3. Neither agree nor disagree4. Agree5. Strongly agree
“The attorney communicated with me regularly.”
1. Strongly disagree2. Disagree3. Neither agree nor disagree4. Agree5. Strongly agree
Copyright © 2018 by OneJustice. All rights reserved.
Open questionsUse these when you do not want to narrow the possible range of response
What was most helpful about the service you received from Legal Aid?
In your opinion, how could the service be improved?
Copyright © 2018 by OneJustice. All rights reserved.
User testingUse qualitative interviews to test the survey with respondents (esp. if complex)
Copyright © 2018 by OneJustice. All rights reserved.
Fieldwork• Internal and external communications
• Changes to organizational processes
• Training and managing staff / volunteers
• During fieldwork, monitoring
• Response rates
• Non-response to specific questions
• Interviewer consistency
• Data management in spreadsheets
• Managing follow-up
Copyright © 2018 by OneJustice. All rights reserved.
Preparing survey data for analysis
Resp. ID Age Office Legal Area
Q1. Listening
Q1 Numeric
Q3. Filed
1 20 A Family S. Agree 5 Filed
2 30 A Family Agree 4 Filed
3 40 B Family Neither 3 Not Filed
4 50 A Family Disagree 2 Refused
5 40 B Family S. Disagree 1 Filed
Copyright © 2018 by OneJustice. All rights reserved.
Analysis of survey dataFrequencies + proportions
Yes filed Not filed Missing Total
Frequency 35 13 2 50
Percentage 70% 26% 4% 100%
Valid % 73% 27% -- 100%
Cross-tabulationsYes filed Not filed Missing Total
Office A 23 (77%) 6 (20%) 1 (3%) 30 (100%)
Office B 12 (60%) 7 (35%) 1 (5%) 20 (100%)
Total 35 (70%) 13 (26%) 2 (4%) 50 (100%)
Copyright © 2018 by OneJustice. All rights reserved.
Analysis of survey data3. Grouping ordered responses
Agree/strongly agree
Neither Disagree/Strongly disagree
Total
Frequency 30 5 15 50
Percentage 60% 10% 30% 100%
4. Ordered responses as numericStatistic Listening
Mean 4.2
SD 1.2
Copyright © 2018 by OneJustice. All rights reserved.
Survey Design Workshop
Copyright © 2018 by OneJustice. All rights reserved.
Thank you! Wrapping upReflections from today