LLEWELLYN J. CORNELIUS, PH.D.UNIVERSITY OF MARYLAND

Using Cultural Competency Assessment Tools as a means of

evaluating the reduction of health disparities among persons of color.

Correspondence: Please address all correspondence to Dr. Llewellyn J. Cornelius, Professor- University of Maryland School of Social Work, 525 W. Redwood Street, Baltimore, MD 21201. Telephone 410-706-7610; Fax 410-706-6046; Email: lcornelius@ssw.umaryland.edu

Context:

The need for a culturally competent workforce has been reflected in a multitude of publications, including the both the Culturally and Linguistically Appropriate Services and the National Partnership for Action recommendations published by the U.S. Department of Health and Human Services’ Office of Minority Health (2002, 2010).

Purpose:

The purpose of this presentation is to provide an overview of how a consumer-based cultural competency inventory was developed, validated and re-validated using Community Based Participatory Research.

4

Cultural Competency Inventory Validity Process Using CBPR

FrameworkRecognize community as a unit of identity.Recognize community as a unit of identity. – –

The unit of identity was the CCAG (The unit of identity was the CCAG (Cultural Competency Advisory Group, a 20-person panel of consumers, therapists, and administrators that advised the Mental Hygiene Administration. The he project required their buy-in and participation.project required their buy-in and participation.

Build on strengths and resources within the Build on strengths and resources within the community.community. – The priorities of the community as – The priorities of the community as reflected by the CCAG leadership was used to reflected by the CCAG leadership was used to guide the development of the instrument.guide the development of the instrument.

Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of research: Assessing partnership approaches to improve public health. Annual Review of Public Health. 19: 173-202Public Health. 19: 173-202

5

Engagement Process Using CBPR Framework (continued)

Facilitate collaborative partnerships in all Facilitate collaborative partnerships in all phases of the research.phases of the research. – The presenter and – The presenter and co-presenters’ role was that of providing co-presenters’ role was that of providing technical assistance in research methods to the technical assistance in research methods to the CCAG, while theirs is that of participating in the CCAG, while theirs is that of participating in the design of the study and the data collection design of the study and the data collection activities.activities.

Integrate knowledge and action for the Integrate knowledge and action for the mutual benefit of all partners.mutual benefit of all partners. – Knowledge – Knowledge provided by the CCAG regarding the historical provided by the CCAG regarding the historical and contemporary mistreatment of ethnic and contemporary mistreatment of ethnic populations was critical to the success of the populations was critical to the success of the project.project.

Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based research: Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health. 19: 173-202Assessing partnership approaches to improve public health. Annual Review of Public Health. 19: 173-202

6

Engagement Process Using CBPR Framework (continued)

Promote a co-learning and empowering Promote a co-learning and empowering process that attends to social inequitiesprocess that attends to social inequities. –. –The CCAG facilitated a process that allowed The CCAG facilitated a process that allowed for the discussion of content that would lead to for the discussion of content that would lead to the empowerment of community members.the empowerment of community members.

Involve a cyclical and iterative process.Involve a cyclical and iterative process. – – The CCAG meets monthly to address the needs The CCAG meets monthly to address the needs the community.the community.

Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of research: Assessing partnership approaches to improve public health. Annual Review of Public Health. 19: 173-202Public Health. 19: 173-202

Validity and Reliability testing process

he following activities were conducted during this period; 1. Literature review to map out the concepts of

cultural competency and satisfaction with the quality of care against the questions in the survey.

2. Redrafting the survey tool based on the literature review.

3. Pilot testing the instrument with 50 respondents 4. Revising the tool based on the pilot test 5. Translating the instrument into Spanish and

Vietnamese.

Validity and Reliability testing process cont’d

he following activities were conducted during this period; 6. Pilot testing the instrument with a sample of

240 respondents in public mental health facilities across the state of Maryland

7. Conducting and computing correlations between responses to survey questions and conducting an exploratory factor analysis to examine the relationship between measures in the instrument.

Validity and Reliability testing process (2001-2002) Literature ReviewOne of the important factors in the design

and implementation of surveys is making sure that each question in an instrument can be tied to some concept or measure.

As such, the team proceeded by conducting a literature review to classify the significant themes in the literature on satisfaction with the quality of care and cultural competency. The team, then, examined the questions in the instrument against these themes, eliminating questions that were not supported by the literature.

Validity and Reliability testing process –Literature Review

The following themes emerged from the literature: Attitude – how consumers feel they are perceived Communication – issues related to language,

being talked with and heard Treatment – use of healing practices, family

involvement and spirituality Personnel – availability of multicultural staff at

various levels Environment – perceptions of feeling welcomed by

staff and agency Outreach – commitment of staff and agency to

engage the community

The Validity and Reliability testing process –Instrument Revision

After eliminating questions that were not supported by the literature, the team used principles recommended by Aday and Cornelius (2006) and Sudman and Bradburn (1982) to fine tune the questions. In particular we focused on: Revising questions that had multiple meanings Examining the questions for readability Shortening questions wherever possible Grouping similar questions together Changing the wording for some similar questions

from “agree” to “disagree” to detect inconsistencies in responses to the similar questions.

Formatting the questionnaire to facilitate response

Validity and Reliability testing process Initial Pilot Test(n=50)

After fine tuning the questions, we conducted a pilot test with a convenience sample to focus solely on questions consumers had regarding the clarity of the questions presented and the survey instructions.

After we completed the initial pilot test, we noted that while we achieved the goal of face validity (that is, the instrument validity based on the logic of the instrument) using experts, consumers and the literature to design the instrument, we realized that we can start down the road of conducting more rigorous studies of the validity and reliability testing of the study, if we had a larger sample.

Validity and Reliability testing process –Second Pilot Test (n=240)

Utilizing a training manual developed by the consultant, nine members of the CCAG were trained to administer the pilot test. The manual addressed issues such as engaging volunteer respondents, communicating purpose and procedures for the pilot test, discussing confidentiality and risk/discomforts, facilitating group testing and managing collected data (Cornelius, 2001). Of the nine survey administrators, three were proficient in Spanish and one in Vietnamese.

Validity and Reliability testing process –Second Pilot test (n=240)

Using claims data from MHP, 13 of the 24 jurisdictions in Maryland, geographically dispersed across the state were selected to assure a balanced regional sampling of respondents of color. These specific jurisdictions also provided the opportunity to sample respondents from urban, suburban and rural areas. Between late January and early April 2002, pilot testing occurred at 30 Psychiatric Programs throughout the state of Maryland.

The empirical examination of the validity and reliability of this instrument followed the administration of the 52 item scale to mental health consumers across the state of Maryland in the January 2002.

Validity and Reliability testing process –Second Pilot test (n=240)

Separate analyses of the validity and reliability of the questionnaire items revealed this 52-item scale that had good psychometric properties (Chronbachs alpha=.91) (Cornelius, Arthur, Booker and Morgan, 2003).

Other analyses (correlations and exploratory factor analyses) suggest that there is room for further fine tuning of this instrument. Finally it some of the items in the instrument also lend themselves to other types of validity testing- for example testing the validity of some of the questions regarding the clinical setting via self reported (the current questions) and interview observed measures (new measures) of the clinic setting.

A second re-test of the instrument

Following this assessment process, the Mental Hygiene Administration completed a separate re-assessment of the tool to determine whether the instrument would provide consistent findings when examined with another sample of consumers. The instrument was found to have similar findings when re-administered.

Plans are currently underway to pilot the assessment tool in an public mental health care setting as a measure of the delivery of mental health services.

Results: The Validity and Reliability testing process

Construct Validity FindingsTABLE 1: Subscale Intercorrelations, Subscale 1 2 3 4 5 6 7 8 1. Awareness of Patients’ Culture subscale .— .68 .76 .38 .42 .54 .14 .69 2. Respectful Behaviors subscale — .46 .52 .54 .68 .08 .163. Language Interpreter Issues subscale .— .29 .31 .41 .22 .51 4. Understanding of Indigenous Practices subscale .— .64 .50 .41 .57 5. Consumer Involvement subscale . — .56 .35 .66 6. Acceptance of Cultural Differences subscale .— .35 .61 7. Community Outreach subscale — .30 8. Patient-Provider-Organization Interactions subscale .—

Results: The Validity and Reliability testing process

Reliability Findings

TABLE 2: Cronbach’s Alpha and Eta-Square Subscale Coefficient Alpha Eta-Square1. Awareness of Patients’ Culture subscale .69 .162. Respectful Behaviors subscale .61 .223. Language Interpreter Issues subscale .08 .224. Understanding of Indigenous Practices subscale .70 .515. Consumer Involvement subscale .72 .456. Acceptance of Cultural Differences subscale .52 .237. Community Outreach subscale .74 .538. Patient-Provider-Organization Interactions .79 .40Overall score .92 .13

Conclusion

The leadership of a consumer led group resulted in the development of an inventory that has good psychometric properties based on the results of the validity and reliability testing of this scale.

Given these findings, it is believed that the overall scale, as well as, one of the subscales can be used to examine consumer perceptions of the cultural competency of their mental health providers. As such, it can be used as a part of a pre-assessment and post-assessment of cultural competency.

References

Aday, LA. and Cornelius, LJ Designing and Conducting Health Surveys. (3rd Edition) (2006). San Francisco, Jossey-Bass.

Arthur, TE, Reeves, IG, Cornelius, LJ, Booker NC, Morgan, O, Brathwaite, J, Tufano, T, Allen, K., Donato, I. “A Consumer Assessment Tool for Cultural Competency Within Maryland’s Public Mental Health System.” Psychiatric Rehabilitation Journal, 2005. Vol 28 (3): 243-250.

Cornelius, LJ, Booker NC, Arthur, TE, Reeves, IG Morgan, O,. “The validity and reliability testing of a consumer base cultural competency inventory.” Research on Social Work Practice, 2004, Vol. 14(3) 201-209.

Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review Israel, B.A. & Schulz, A.J., Parker, E.A., Becker, A.B. (1998). Review of community-based research: Assessing partnership approaches to of community-based research: Assessing partnership approaches to improve public health. improve public health. Annual Review of Public HealthAnnual Review of Public Health. 19: 173-. 19: 173-202202

Sudman, S. Bradburn, N. (1982) Asking Questions.