Summit 2010 Measures Aidala

8/9/2019 Summit 2010 Measures Aidala

1/25

Wheres the Evidence?Wheres the Evidence?Measurement ChallengesMeasurement Challenges

Angela A.Angela A. AidalaAidala

Columbia University Mailman School of Public HealthColumbia University Mailman School of Public Health

Columbia Center for Homelessness Prevention StudiesColumbia Center for Homelessness Prevention Studies

HOUSING AND HIV/AIDS RESEARCH SUMMIT VHOUSING AND HIV/AIDS RESEARCH SUMMIT V

June 2, 2010June 2, 2010 Toronto,OntarioToronto,Ontario


2/25

Presentation GoalsPresentation Goals

Discuss data collection methods and tools for:Discuss data collection methods and tools for:

Describing/ documenting programsDescribing/ documenting programs

What is the program about? What does it do?What is the program about? What does it do?

Describing/ documenting contextsDescribing/ documenting contexts

What are risk promoting or protective contexts?What are risk promoting or protective contexts?

Answering process questionsAnswering process questions

How did it happen? How does it operate?How did it happen? How does it operate?

Answering outcomes questionsAnswering outcomes questions

So what? What difference did it make in the lives ofSo what? What difference did it make in the lives of

individuals, families, communities?individuals, families, communities?


3/25

Are you just pissing and moaning, or can you verify what youre saying with data?

Credible EvidenceCredible Evidence


4/25

Collecting DataCollecting Data

Different data collection strategies areappropriate for answering research questions

Surveys / interviews - structured conversation using

a questionnaire or interview guide

Focus groups (group interviews)

Records review (staff reports)

Administrative data (surveillance indicators)

Biomedical assessment (x-rays, blood tests)

Ethnography, observational analysis (being there)

Environmental assessments (material, social context)


5/25

MeasuresMeasures

Most of what we want to study cannot be directly seen,heard, touched, etc.

- Racism, stable housing, good health

Need procedures to define concepts so that they can be

measured

- Stable housing = same address 6+ months

Need to be RELIABLE (consistent result when use tool)

Need to be VALID ( is true measure of phenomena)

Need to be appropriate to answer research question

Need to be appropriate for different populations


6/25

What Should be Measured?What Should be Measured?

Contexts: Social, Economic, Political, Cultural, ServiceContexts: Social, Economic, Political, Cultural, Service

Census dataCensus data

City/county dataCity/county data

HIV surveillanceHIV surveillance

Service directoriesService directories

Newspaper archivesNewspaper archives

Informant interviewsInformant interviews

Need to consider: Peoples perceptions and experiencesNeed to consider: Peoples perceptions and experiences

Useful tools: HUDuser.gov, AmericanUseful tools: HUDuser.gov, American FactFinderFactFinder


7/25

Ex: Distressed NeighborhoodEx: Distressed Neighborhood

Neighborhood operationally defined as blockgroup distressed indicated by summary %

households below 75% of the poverty line

vacant housing units

female-headed households with children < 6yrs

unemployment rate

public assistance income

crimes against persons per 1,000 population

property crime per 1,000 population

Lee (2009). Factors Associated With Departure From Supported Independent Living Programs


8/25


Process data: Information about how a programProcess data: Information about how a programworks and lessons learned implementing the programworks and lessons learned implementing the programthat could guide replication or expansionthat could guide replication or expansion

Routine agency documentsRoutine agency documents

Informant interviewsInformant interviews

Meeting notesMeeting notes

ObservationsObservations

Useful strategy: MonthUseful strategy: Month--byby--month chronologymonth chronology

Useful tool: Standard form for meeting notesUseful tool: Standard form for meeting notes


9/25

Ex: Record of Issues and DecisionsEx: Record of Issues and Decisions


10/25

Outcome Indicators: Specific items of informationOutcome Indicators: Specific items of informationthat track a programs success on outcomesthat track a programs success on outcomes

Levels of outcomes can focus on any of:Levels of outcomes can focus on any of:

---- Individual (client)Individual (client)---- Agency, organization, programAgency, organization, program

---- Group, social network (family, local club scene)Group, social network (family, local club scene)

---- Service SystemService System---- BroaderCommunityBroaderCommunity


10


11/25

Outcome indicators can focus on:Outcome indicators can focus on:


11

Desired Change: Types ofOutcomes:

Attitudes Improved self-efficacy

Perceptions Increased perception of need

Knowledge Greater awareness of services

Skills Improved problem solving

Behaviors Increased condom use

Conditions Reduction of housing instability


12/25

12

Interviewing Clients forInterviewing Clients forService Delivery AND ResearchService Delivery AND Research

Does Not Mean:Does Not Mean:

Staff cannot have theirStaff cannot have their

own styleown style Information restricted toInformation restricted to

check listscheck lists

All information must beAll information must be

obtained at one timeobtained at one time Staff cannot establishStaff cannot establish

rapport with clientsrapport with clients

Does Mean:Does Mean:

Intent of questions must beIntent of questions must be

understoodunderstood

Systematic way to gatherSystematic way to gatherinformationinformation

Information about the sameInformation about the same

client must be linkedclient must be linked

Interviewers cannot leadInterviewers cannot leadclients to answerclients to answer


13/25


14/25

14

Some RulesSome Rules

There should be good reason for asking eachThere should be good reason for asking eachquestionquestion

Use established questions and scales whereUse established questions and scales whereappropriateappropriate e.g. SFe.g. SF--12, CDQ12, CDQ

Begin with topics the client might want to discussBegin with topics the client might want to discussNOT demographicsNOT demographics

Provide interviewer instructions on formProvide interviewer instructions on formincluding skip directionsincluding skip directions

For closed questions make sure all alternativesFor closed questions make sure all alternativesare presentedare presented

No double barrel questionsNo double barrel questions

Shortest is not always best !Shortest is not always best !


15/25

HOPWA INTAKE FORMDate:_______________________ Intake Worker:____________________________________

Referral Source:_______________________________________________________________

Legal Name:________________________________________ Other Name_______________

Phone:_________________________________Beeper:_______________________________

Current Address:______________________________________________________________

____________________________________________________________________________

Last Known Address___________________________________________________________

____________________________________________________________________________

Date of Birth:_________________ Age:_______ Place of Birth: City____________________

State:________________ Country:__________________ U. S. Citizen: Y N

Sex:________________ Sexual Orientation:_______________ SS#____________________

Race:_____________________ Ethnicity:___________________ Religion:_______________

Emergency Contact:

Name/Relationship: _____________________________________ Phone:________________

PRESENTING PROBLEM AND SERVICES REQUEST:

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

Other agency involvement:______________________________________________________

____________________________________________________________________________

Services Requested (circle)

shower food clothes shelter medical legal birth control/ condoms

counseling help with family problems GED job training identification

public assistance Medicaid permanent housing HIV testing HIV education


16/25

Client ID # |___|___|___|___|

HARBOR OF HOPE PROGRAM ASSESSMENT FORM

Todays Date:____ /____/____ Interviewer:________________________

mm dd yyyy

1. What is your name________________________________________________

1a. Preferred name________________________________________________

2. What brings you here today? ( State in clients own words):

_____________________________________________________________________

_____________________________________________________________________

_____________________________________________________________________

_____________________________________________________________________

_____________________________________________________________________

Services Requested (circle all that apply)

01...Shower 06...Birth control 11Job Training 02...Food 07Condoms 12Public Assistance

03...Clothes 08... Counseling, MH 13...Medical services

04...Emergy Shelter 09AOD services 14...Medicaid

05...Permt Housing 10...GED, education 15HIV testing

Well talk more about these issues later in the questionnaire but first some

background questions .


17/25

17

TYPICAL QUESTION ABOUT CLIENT HEALTH:

Health:______________________________________________________________________________

_____________________________________________________________________________________

_____________________________________________________________________________________

STANDARDIZED HEALTH MEASURE:

Next are some questions about your health at the present time.

1. In general would you say your health is. . .

Excellent. . . . . . . . . . . 1

Very Good. . . . . . . . . . 2

Good. . . . . . . . . . . . . . 3

Fair . . . . . . . . . . . . . . . 4

Poor. . . . . . . . . . . . . . . 5

2. The following items are about activities you might do during a typical day. Does your health

now limit you in these activities . . .


18/25

18

Indicators: Physical HealthIndicators: Physical Health

CD4Count provide categories (i.e. belowCD4Count provide categories (i.e. below200, above 500)200, above 500)

Viral loadViral load undetectableundetectable v.v. detectabledetectable

Score on health functioning measure:Score on health functioning measure:MOSMOS SF12SF12

General health question: Health nowGeneral health question: Health nowexcellent, very good, good, fair, or poor.excellent, very good, good, fair, or poor.


19/25

19

Indicators: Mental HealthIndicators: Mental Health

Client underClient under--recognition of need for mentalrecognition of need for mentalhealth serviceshealth services

Systematic screening v. seems depressedSystematic screening v. seems depressed

Use standardized measure as well as selfUse standardized measure as well as self--report of problemsreport of problems

Diagnostic screeners (Diagnostic screeners (e.ge.g CDQ) provide moreCDQ) provide more

information than symptom checklistsinformation than symptom checklists


20/25

20

Engagement with Medical CareEngagement with Medical Care

Usual source of medical care can beUsual source of medical care can beambiguousambiguous

Need also ask: How many visits for HIV careNeed also ask: How many visits for HIV care

from a doctor or other medical provider didfrom a doctor or other medical provider didyou have in the past 6 months?you have in the past 6 months?

ER visit, hospitalized over night or longer inER visit, hospitalized over night or longer inthe past 6 monthsthe past 6 months

Stopped going to the doctor for any reasonStopped going to the doctor for any reasonfor 6 months or longerfor 6 months or longer


21/25

21

ASKING ABOUT SENSITIVE TOPICSASKING ABOUT SENSITIVE TOPICS

Provide brief introduction to question and giveProvide brief introduction to question and giveassurance of confidentialityassurance of confidentiality

Ask whether client hasAsk whether client has everever engaged inengaged inbehavior before asking about current behaviorbehavior before asking about current behavior

Ask how often or when was the first ratherAsk how often or when was the first ratherthan did you everthan did you ever

Staff discomfort with questions will make clientStaff discomfort with questions will make clientuncomfortableuncomfortable


22/25

22

5. Have yo u e ver use d any o f the follow ing, eve n one tim e...Go dow n entire list, then go back a nd for a ny YES ask : Did you u se (drug) in the pa st six m onth s?

Ever usedUse d pas t sixmonths

SUBSTA NCE Yes No Yes No

1. M arijua na, has his h 1 0 1 0

2. Powd ered C ocain e 1 0 1 0

3. C rac k (roc k ) 1 0 1 0

4. H eroin or s peedb all 1 0 1 0

5. A m phetem ines o r m etha nphetem ines 1 0 1 0

6. Inhala nts , p oppe rs 1 0 1 0

7. A ny other dru g us ed regular ly,3+ t imes wk o r more often:

Drug: _____________________

1 0 1 0

5. Have yo u e ver use d any o f the follow ing, eve n one tim e...Go dow n entire list, then go back a nd for a ny YES ask : Did you u se (drug) in the pa st six m onth s?

Ever usedUse d pas t sixmonths

SUBSTA NCE Yes No Yes No

1. M arijua na, has his h 1 0 1 0

2. Powd ered C ocain e 1 0 1 0

3. C rac k (roc k ) 1 0 1 0

4. H eroin or s peedb all 1 0 1 0

5. A m phetem ines o r m etha nphetem ines 1 0 1 0

6. Inhala nts , p oppe rs 1 0 1 0

7. A ny other dru g us ed regular ly,3+ t imes wk o r more often:

Drug: _____________________

1 0 1 0

E . S U B S T A N C E U S E

M a n y p e o p le h a v e e x p e r ie n c e s w i h a lc o h o l a n d u s e o f d if f e r e n t s u b s t a n c e s . W e w o u ld l i e

to k n o w a b o u t yo u r e x p e r ie n c e a s p a r t o f y o u r h e a l t h p r o f i le . I d li k e to r e m in d y o u t h a t

e v e r y t h in g y o u s a y is s t r ic t ly c o n f id e n t ia l a n d p r o t e c t ed

E . S U B S T A N C E U S E


23/25

23

DATA QUALITY ASSURANCE

All persons using forms must be trainedAll persons using forms must be trained

Dont assume knowledge or comfort level ofDont assume knowledge or comfort level ofstaffstaff

Use training documents e.g.Use training documents e.g. QxQQxQ

Pilot test data collection with consumersPilot test data collection with consumers

Review completed forms for clarity andReview completed forms for clarity andcompletenesscompleteness


24/25

My question is, Are we making a difference?


25/25

25

RESOURCES FOR MEASURESRESOURCES FOR MEASURES

Tools for HOPWA Outcome Based EvaluationTools for HOPWA Outcome Based Evaluationhttp://www.aidshousing.org

Community ToolboxCommunity Toolbox -- Resources for HealthyResources for HealthyCommunitiesCommunities http://ctb.ku.edu/en/Default.htm

Columbia Center for Homelessness PreventionColumbia Center for Homelessness PreventionStudiesStudies -- Measures LibraryMeasures Library http://www.columbia-chps.org/resources/Homelessness-and-Housing-Measures

MOSMOS--SF12 v2SF12 v2 http://www.sfhttp://www.sf--36.org/tools/sf12.shtml36.org/tools/sf12.shtml

CDQ & Training ManualCDQ & Training Manual [email protected]@columbia.edu

HRSA, CDC, SAMHSA online guidesHRSA, CDC, SAMHSA online guides

Documents

Summit 2010 Measures Aidala