MHHSC PROGRAM

Constructing a Cross-Site Evaluation of Ethnic Minority

HIV Mental Health Services

The Mental Health HIV Services Collaborative (MHHSC) Program

Michael Costa, AbtBarbara J. Silver, CMHSMaria Madison, AbtTandiwe Njobe, AbtGabriela Garcia, Abt

MHHSC PROGRAM

Presentation Goals

Describe the MHHSC Program Components Context Relevance

Convey: The process of this collaborative. utilization-

focused evaluation Accomplishments to date (products) Uses of the outputs of products

MHHSC PROGRAM

Substance Abuse Mental Health Services Administration

(SAMHSA)

Center for Mental Health Services (CMHS)Center for Substance Abuse Prevention (CSAP)Center for Substance Abuse Treatment (CSAT)

MHHSC PROGRAM

SAMHSA HIV/AIDS HISTORY

Prior to 2001: CSAT funds targeted capacity/HIV/AIDS Substance Abuse treatment programs for African American, Hispanic/Latino, and other racial/ethnic minorities.2001: CMHS funds a similar targeted/expanded capacity program for community based organizations (CBOs) serving African American, Hispanic/Latino, and other racial/ethnic minorities.

MHHSC PROGRAM

HIV Infection among People with Severe Mental Illness

Across all published studies, the rate of HIV infection among psychiatric patients is 10%, 25 times higher than that of the general population.

Cournos & McKinnon, 1997;Krakow et al., 1998;Rosenberg et al., 2001

HIV Prevalence in Psychiatric and General Populations

10

0.402468

1012

Psychiatric General

MHHSC PROGRAM

Number of partners Number of risky or anonymous partners Frequency of sex trading Rates of coerced sex

McKinnon et.al., 1996, 1999

COMPARED TO GENERAL POPULATION, PATIENTS HAVE FEWER COMPARED TO GENERAL POPULATION, PATIENTS HAVE FEWER EPISODES OF SEX WITH A PARTNER, BUT THEY HAVE:EPISODES OF SEX WITH A PARTNER, BUT THEY HAVE:

Sexual Risk Behavior Among People With Severe Mental Illness:

MHHSC PROGRAM

Elevated risk for HIV infection in psychiatricPatientsRisk factors: Alcohol and other drug use Unsafe sex Environmental circumstances (poverty, institutionalization, etc.)

Substance use is associated with both psychiatric symptoms and HIV risk

Psychiatric Disorders and Risk for HIV Infection

MHHSC PROGRAM

MHHSC Program

21 Mental Health Service Sites – CBOs at least 2 years experience in behavioral

health care services MH Centers, Substance Abuse facilities, Primary Health Care &/or HIV/AIDS clinics

Abt Associates, Inc. = Coordinating Center

MHHSC PROGRAM

MHHSC ProgramCongressional requirement (CBC & CHC): provide these new HIV/AIDS-related mental health services in both traditional and non-traditional settings.Funding for mental health treatment services and related case management only.However, grantees are required to develop comprehensive integrated individual treatment plans and monitor primary and substance use treatment.

MHHSC PROGRAM

WHO ARE THE SERVICE SITES?

New HIV/AIDS-Related Services New services (no prior HIV/AIDS-related MH services)– 5 sites Expanded services – 16 sites

Service Delivery Settings Traditional (primarily clinic-based) – 13 sites Non-traditional (e.g., mobile treatment, ) – 1 site Both settings – 8 sites

Target Populations African American -19 sites Hispanic/Latino – 14 sites Haitian – 1 site Native American – 1 site

MHHSC PROGRAM

Demographics – Gender

61.90%

37.10%

0.90%0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Male Female Other

MHHSC PROGRAM

Hispanic Latino/a = 30.65%

Demographics – Race / Ethnicity

Native American3%

Asian0%Pacific Islander

0%White17%

Other Race30%

African American/ Black50%

MHHSC PROGRAM

Demographics - Age

Age Category PercentLess than 20 years 1.72%20 – 24 years 4.68%25 – 29 years 7.41%30 – 34 years 10.76%35 – 40 years 20.83%40 – 44 years 22.78%45 – 50 years 14.51%50 – 54 years 8.74%55 years and older 6.01%

MHHSC PROGRAM

DSM IV Diagnoses

27.21%

5.39%

50.51%

16.89%

0%

10%

20%

30%

40%

50%

60%

Single Dx Dual Dx Triple Dx More than 3 Dx

MHHSC PROGRAM

DSM IV Diagnoses Categories

9.44%13.88%

6.63%

45.71%

24.33%

6%

12%

18%

24%

30%

36%

42%

48%

AdjustmentDisorder

AnxietyDisorder

BipolarDisorder

DepressiveDisorder

OtherIncluding HIV

Dementia

MHHSC PROGRAM

MHHSC Program Goals

ExpandEffectiveCulturally CompetentMental Health ServicesFor PLWHIVIn Minority Communities

MHHSC PROGRAM

MHHSC Capacity Building

The Coordinating Center provides technical assistance to grantees:Local and regional trainings (e.g., ethics, neuropsychology, cultural competence)Expert speakers at national meetings on topics of mental health and psychiatry, gender issues, consumer involvement, etc.Assistance with data collection and management

MHHSC PROGRAM

CULTURAL COMPETENCE

Strategies to Promote Culturally-Competence Service Delivery Demographically similar staffing as target population(s) Treatment facilities reflect cultural interests of target

group(s) (e.g., artwork/decor, waiting room music, etc.) Diversity/cultural competence training Consumer feedback

This is being addressed in great detail by the MHHSC Cultural Competence Subcommittee

MHHSC PROGRAM

MHHSC Utilization-Focused Cross-site Evaluation

Clinically/Programmatically Relevant Evaluation meet the needs of the clinical and other program staff in

their efforts to better serve their clients

The MHHSC cross-site evaluation is voluntary, except for submission of required aggregate data critical that the cross-site evaluation be clinically and

programmatically relevant and valuable, otherwise no site would agree to participate

MHHSC PROGRAM

Evaluation as a Pain in the NeckMental health/social service providers’ historical experience with evaluation Intrusive reporting requirement Necessary to satisfy Local, State and Federal funding

requirements.

Often data are never reported back to programsConcerns that data will used to make them look bad or draw inappropriate comparisons between sites and providers

MHHSC PROGRAM

MHHSC X-Site Evaluation Collaborative Process

Engaging the key stakeholdersLocal site-specific evaluatorsLocal site cliniciansProgram AdministratorsConsumer Advisory Boards (CABS)

MHHSC PROGRAM

MHHSC X-Site Evaluation Collaborative Process

EvaluationSubcommittee

(ESC)

Logic ModelWorkgroup

Eval/ClinicianCommunication

Workgroup

Site ProgramLogic Models

Site-SpecificEvaluation Summaries

Determination ofEvaluation Foci

Across Sites

MHHSC PROGRAM

MHHSC X-Site Evaluation Collaborative Process

Face-to-FaceMeeting w/MHHSC

Stakeholders

X-Site CandidateEvaluation

Foci

MHHSC PROGRAM

X-Site Candidate Evaluation FociFoci by Quantitative and Qualitative Methods

0

5

10

15

20

25

Freq

of M

etho

ds A

cros

s Si

te P

lans

Quantitative Methods

Qualitative Methods

MHHSC PROGRAM

Quantitative Methods Used Across SitesQuantitative Methods

0

5

10

15

20

25

Clie

ntSa

tisfa

ctio

nSu

rvey

Adm

inis

trat

ive

Clie

nt C

onta

ctD

ata

Abs

trac

tion

CM

HS

GPR

A(w

ith a

ndw

ithou

tau

gmen

tatio

n)

Surv

eyD

evel

oped

by

Eval

uato

r

Clin

ical

Rec

ords

Abs

trac

tion

Inta

ke a

ndA

sses

smen

tTo

ol

Freq

Acr

oss

Site

Pla

ns Quantitative Methods

MHHSC PROGRAM

Qualitative Methods Used Across SitesQualitative Methods

0

5

10

15

20

25

Focus Groups Interviews Observation ClinicalRecordReview

Freq

Acr

oss

Site

Pla

ns

MHHSC PROGRAM

MHHSC X-Site Evaluation Collaborative Process

Breakout Groups(BOGs)

EvaluationDesign

Workgroup

MHHSCNationalMeeting

ConsumerNetwork

Committee

X-Site CandidateEvaluation

Foci

Face-to-FaceMeeting w/MHHSC

Stakeholders

MHHSC PROGRAM

MHHSC X-Site Evaluation Collaborative Process

Next Steps

Pilot Test

Instruments/Protocols

Final DraftField X-SiteEvaluation

Pray Everything Goes As Planned

Instruments/ProtocolsBeta Version

X-Site DesignSite Review

MHHSC PROGRAM

Final Cross Site Foci

Four main domains* of interest across sites:Mental HealthClient SatisfactionClient Retention and Service UtilizationQuality of Life & Medical Health

*Cultural Competence appears in all domains.

MHHSC PROGRAM

Common Questions of Interest

Who is being served? What are the barriers to care and to services for the target population? How do programs overcome these barriers? What services are being used by the target population? Are the services being provided in a culturally competent manner?

MHHSC PROGRAM

Mental HealthGiven the target population being served, what are the Prevailing mental health diagnoses?Co-occurring disorders? (e.g., substance use)Changes in mental health symptoms?Disorders that may be more prevalent with HIV

positive status?Physiological HIV disease factors that

contribute to mental health symptoms?

MHHSC PROGRAM

Client Satisfaction

Questions relating to care received by clients.What care is being received?What is the frequency of care?What is the level of client involvement in care?What is the setting in which care is received? Is the client satisfied with the care?

MHHSC PROGRAM

Client Retention & Service Utilization

Client RetentionHow many cases become active/inactive in a

given time period?What is the site definition for active/ inactive

clients?What are the site policies for case closing? What

reasons are given for case closings?What retention strategies have been effective

with the target population?

MHHSC PROGRAM

Client Retention & Service Utilization - continued.

Service UtilizationWhat is the type, frequency and duration of

services used?How are services provided?

Referrals in and out of programAgency linkages - in-house and external - to other

providersService setting – traditional/ non-traditional

MHHSC PROGRAM

Quality of LifeHow does the quality of life status and general medical health of a client impact uponTreatment compliance (medications, ITP)?Response to treatment?Client retention? Service utilization?

How does quality of life status and general medical health change over time with treatment?

MHHSC PROGRAM

Methods & Indicators1. CLIENT LEVEL INSTRUMENT

confidential – unique client id

Demographics Social supports Acculturation Substance use and risk behavior Trauma Medical health Medication adherence Clinicians report

MHHSC PROGRAM

Methods & Indicators2. CLIENT SATISFACTION SURVEY

anonymous

Client characteristics Service utilization Care

Client involvement in care Access/ barriers to care Cultural competence in care Overall satisfaction with care

MHHSC PROGRAM

Methods & Indicators3. FOCUS GROUPS

with clients on site

Focus groups will provide qualitative backdrop to analyze quantitative data from client satisfaction survey and client instrument

Types of services used Satisfaction with services Barriers to care Cultural competency

MHHSC PROGRAM

Methods & Indicators4. SITE VISIT DATA COLLECTION

MHHSC Coordinating Center Staff

Continuum of services Location of site

Geographic setting Site community

Service setting (traditional / non traditional) Client retention strategies Site activities to overcome identified barriers to

mental health care

MHHSC PROGRAM

Utility of Cross-Site Analyses From the Sites’ Perspective

Site buy-in has been an on-going processClinicians and local evaluators participate in the evaluation design work group They took ownership of the evaluation designMade decisions on utility of collecting certain types of data across sites

MHHSC PROGRAM

Client Focused Domain

Client CharacteristicsClients’ Presenting DiagnosisBarriers/Access to Care (enhance/maintain Client’s Quality of Life/Health)

MHHSC PROGRAM

Program Context Domains

Program StructureCBO VS Large System of care Staffing patterns (FT, PT)

Sustainability EffortsPlan in place

MHHSC PROGRAM

Client Focused DomainsClient Characteristics Site UsesSystematic documentation type of clinical services offered to what target population across the sites

Tailor services to specific populations

Identify (ID) staff training & resource needs

Determine success or not in reaching target population

ID changing client populationsAbility to pool data across all sites by client characteristics

Identify types of clients (positively) impacted by type of treatment

Learn from sites’ experiencesID promising practices

MHHSC PROGRAM

Client Focused Domains (cont.)Mental Health Site UsesExamine patterns/changes in symptomatology. Are symptoms unique to a given population

Better assess clientsDetermine client case-mixDocument increased client needs

Gain insight on co-occurring disorders

Better understand client needsTailor services offeredInform client retention strategies

Examine patterns of mental health disorders across sites

Understand impact of servicesInform site staff on cultural

competence issues

MHHSC PROGRAM

Client Focused Domains (cont.)

Barriers / Access to Care Site Uses

Identify barriers to care specific to target population

Improve engagementImprove retentionID methods for overcoming

stigmaDetermine strategies for inter-

agency collaborationGain insight into cultural

competence issues

Gain better understanding of ease of access to mental health services

MHHSC PROGRAM

Program Context DomainsProgram Structure Site UsesUnderstand contexts in which other sites operate

Determine relevance/applicability of findings across sites

Identify replicable program components

Understand uniqueness of site challenges

Learn from other sites' experiences about ways to overcome barriers to care

Identify promising practices Create a menu of tools to address

barriers

MHHSC PROGRAM

Program Context Domains (cont.)

Site Program Structure (cont’d)

Site Uses

Understand differences in the range of program services

Identify gaps in servicesIdentify staffing/resource needsInform program expansion and

sustainability Identify promising practices in client referrals

Improve client treatment fidelityImprove client retentionCreate a menu of tools to address

barriers to care

MHHSC PROGRAM

Program Context Domains (cont.)

Sustainability Site UsesExamine data on services provided to support funding opportunities

Develop sustainability plans at individual site or cross-site level if similarity in client needs

Develop plan to target specific funding sources

MHHSC PROGRAM

Purposes/Uses of Data

Cross-site analysis Attention to differences across sites, client

characteristics Careful & appropriate Nuanced, not reductionist Program context critical Use of qualitative and program-level data (e.g.,

differences in resources/capacity available)

MHHSC PROGRAM

Purposes/Uses of Data (Recap)UTILIZATION-FOCUSED EVALUATIONData/analysis to be provided to sites User friendly feedback A timely manner For program improvement /development of

better intervention strategiesNOT REPORT CARDS Individual clinicians Individual sites