204 http://ap.psychiatryonline.org Academic Psychiatry, 28:3, Fall 2004

Addiction Training Scale: Pilot Study of aSelf-Report Evaluation Tool for Psychiatry Residents

S. Pirzada Sattar, M.D.James Madison, Ph.D.

Ronald J. Markert, Ph.D.Subhash C. Bhatia, M.D.

Frederick Petty, Ph.D., M.D.

Objective: Alcohol and drug dependence disorders have become common public healthhazards. Psychiatrists encounter these problems in a major portion of their patients. How-ever, recent data suggest that their training does not provide them the confidence to treatthese disorders. Current methods of evaluating residents fail to adequately ascertain thelack of confidence in substance abuse training. Here, we present the Addiction TrainingScale (ATS) that we developed to help trainers identify deficits in residents’ substanceabuse training. Method: We developed the ATS and conducted a pilot study with the psy-chiatry residents at the Creighton University Department of Psychiatry, to test the validityof the ATS as a self-report evaluation tool to measure the level of psychiatry residents’ pre-paredness in treating substance abuse disorders. Results: Our results suggest that theATS is related to the confidence and preparedness that residents express in their ability totreat substance abuse problems. Conclusion: The ATS may be beneficial in assessing psy-chiatry residents’ substance abuse training and identifying deficits, which may be ad-dressed during training. (Academic Psychiatry 2004; 28:204–208)

Dr. Sattar is Assistant Professor of Psychiatry and AssociateDirector of Residency Training at Creighton University Schoolof Medicine, Omaha, Nebraska, and Staff Psychiatrist at theOmaha VA Medical Center, Omaha, Nebraska. Dr. Madison isAssistant Professor of Psychiatry at Creighton University Schoolof Medicine, Omaha, Nebraska. Dr. Markert is Professor ofMedical Education and Director, Center for Medical Educationat Creighton University School of Medicine, Omaha, Nebraska.Dr. Bhatia is Professor of Psychiatry at Creighton UniversitySchool of Medicine, Omaha, Nebraska, and Chief of MentalHealth and Behavioral Science at the Omaha VA Medical Center,Omaha, Nebraska. Dr. Petty is Vice Chairman for Research andProfessor of Psychiatry at Creighton University School of Med-icine, Omaha, Nebraska, and Director of Psychiatric Research atthe Omaha VA Medical Center, Omaha, Nebraska. Address cor-respondence to Dr. Sattar, 3528 Dodge Street, Omaha, NE 68131;syed.sattar@med.va.gov (E-mail).

Copyright � 2004 Academic Psychiatry.

Alcohol and drug dependence-related disordersare increasingly recognized as major public

health problems. Alcohol abuse and dependence arewell-documented causes of serious social, legal, eco-nomic, and health complications. One in every 10deaths in the United States is related to alcohol, and20% of the total national health expenditure for hos-pital care is spent on alcohol-related illnesses (1). Inaddition, alcohol and drug abuse have been identi-fied as common factors in marital conflicts and familydysfunction and may account for an estimated $70billion annually in time lost from work (2). Hyper-tension, cancer, and gastrointestinal, liver, cardiac,and psychiatric diseases have been linked to alcoholand drug abuse (3). Estimates suggest that as manyas 60% of psychiatric patients may have coexistingalcohol and drug abuse problems (4). Further, mostaddictive disorders are frequently underdiagnosed(5). While several factors influence this, one may be

SATTAR ET AL.

Academic Psychiatry, 28:3, Fall 2004 http://ap.psychiatryonline.org 205

residents’ perceived lack of preparedness to treatthese disorders (6).

Therefore, assessing and improving psychiatrists’level of preparedness to diagnose and treat alcoholand drug abuse might have an impact on their abilityto provide adequate medical and psychiatric care tothis patient population.

While several educational and training programshave been developed to better teach psychiatrists andprepare them to treat patients with alcohol and drugrelated problems, studies suggest that their long-termimpact may be limited (7). Further, while junior staffmay be justified in reporting a lack of preparednessto diagnose and treat substance related disorders (8),graduating psychiatry residents reporting a lack ofpreparedness to treat these disorders is a cause forconcern (6).

The Accreditation Council for Graduate MedicalEducation (ACGME) currently requires that psychi-atry residents receive at least 1 month of full-time,supervised evaluation and clinical management ofpatients, either in an inpatient and/or outpatient set-ting and familiarity with rehabilitation and self-helpgroups. This may occur as part of an inpatient or out-patient requirement (8). Although most programsmeet this 1-month mandatory training requirement,most programs offer residents 2 months of addictionstraining during residency, with diversity in timingsand settings (8). During this period, however, mostprograms rely on one key supervisor to educate andtrain residents and evaluate their training (8). Currentmethods of evaluating residents’ training are limitedto faculty observations of clinical performance (9).This single method may not be an optimal gauge ofresidents’ competency to treat substance abuse dis-orders (10). The Psychiatry Residents in Training Ex-amination (PRITE) is a standardized testing tool thathas been used to assess competencies in psychiatrytraining. The PRITE is a standardized examinationtaken each year by all residents. Besides addressinggeneral psychiatric disorders and treatment of thesedisorders, the PRITE contains a variable number ofquestions on addictive disorders. However, this ex-amination and other similar standardized tests mayhave limited ability to consistently and accuratelyidentify deficits in residents’ substance abuse train-ing. A recent report suggesting that graduating resi-dents, after four years of training, felt less than pre-pared to treat addictive disorders is an example of

how the PRITE may have failed to identify this issuefor these residents during training (6). This impliesthat current evaluation measures might not be suc-cessful in identifying residents’ lack of confidence intheir addiction psychiatry training. There are fewother options available to clinical faculty to determineresidents’ competence in this area.

In a recent survey of graduating psychiatry resi-dents, Blumenthal et al. (6) reported that psychiatryresidents expressed preparedness to treat most psy-chiatric disorders. However, when asked about theirpreparedness to treat substance abuse disorders, thenumber of residents expressing preparedness de-creased significantly (6). Blumenthal et al.’s report (6)of this discrepancy was based on one survey questionon substance abuse training. Even though the resi-dents’ response to this one question suggested defi-cits in substance abuse training, a single question can-not provide a complete picture of the residents’preparedness to treat this complex group of disor-ders. Substance abuse training includes educationabout a spectrum of disorders ranging from those as-sociated with legal drugs, such as alcohol and nico-tine, to illegal drugs, such as heroin, cocaine, andmarijuana, as well as abuse of prescription drugs.Each drug requires unique knowledge that mayprove critical for physicians in providing appropriatemanagement and treatment. Further, treating sub-stance abuse involves education of and training inother complex issues regarding a patient’s socialstructure and support, coexisting medical and psy-chiatric conditions, pharmacological and nonphar-macological therapies, and an understanding ofstructured living and self-help groups.

In an effort to provide us with another mecha-nism to assess residents’ preparedness to treat addic-tive disorders, we developed the Addiction TrainingScale (ATS). This is a self-reported questionnaire,which can be completed by residents in a few minutesand will help identify areas where the residents them-selves believe improvement is necessary. We also con-ducted a pilot study of the ATS to assess its validity.

METHOD

The authors developed the ATS as a self-report toolto yield more detailed information about residents’preparedness to treat substance abuse disorders.Training faculty with expertise in the substance abuse

ADDICTION TRAINING SCALE PILOT STUDY

206 http://ap.psychiatryonline.org Academic Psychiatry, 28:3, Fall 2004

field (comprised of six faculty members with fellow-ship training and/or certification by the American So-ciety of Addiction Medicine AM or American Boardof Psychiatry and Neurology PN in addiction psy-chiatry) proposed 16 items that evaluated confidencein both general knowledge and in practice areas be-lieved to be relevant to successful clinical work withthis population. The goal of the research team (con-sisting of three addiction training faculty membersand two psychologists, including a biostatistician)was to develop a questionnaire that was internallyconsistent and similar to the general question used inthe survey by Blumenthal et al. (6) The 16 items wereclustered together by content, and subjects wereasked to measure each item on a 5-point Likert scale(“no knowledge” � 1; “very knowledgeable” � 5).The items asked specific questions about residents’knowledge of self-help groups, pharmacotherapiesfor addictive disorders, alternatives to pharmaco-therapies for addictive disorders, various psycho-therapies for addictive disorders, and structured liv-ing programs. The questionnaire also asked residentsto rate their knowledge (“no knowledge” � 1; “veryknowledgeable” � 5) of treating patients with alco-hol, benzodiazepine, barbiturate, amphetamine, andcocaine addiction. Residents were also asked abouttheir preparedness to work with, treat, and providenonpharmacologic therapy to patients with activesubstance abuse or dual diagnosis.

By using the 16-item questionnaire, the ATS is notonly multifaceted in providing more information, butalso provides more validity when compared to thesingle question method used by Blumenthal et al. (6)The ATS yielded more detailed information about thecomfort level of residents treating patients with sub-stance abuse disorders.

Three experimental hypotheses were tested: 1)whether items on the new questionnaire would cor-relate with each other and with Blumenthal et al.’s (6)general question regarding comfort in diagnosingand treating substance abuse and dependency (inter-nal consistency), 2) whether factor analysis of thequestionnaire would yield more than one factor (mul-tidimensionality), and 3) whether residents withmore training in substance abuse or more years ofgeneral psychiatry training would score higher on thenew questionnaire than residents without substanceabuse training or those with fewer years of generalpsychiatry training (criterion validity).

The subjects were 21 psychiatry residents whoagreed to participate according to the procedures ofthe Creighton University Institutional Review Board.Questionnaires were completed anonymously. Nineresidents had completed a specific rotation in thetreatment of substance abuse and alcohol related dis-orders. Residents ranged from years 1 to 5 in theiroverall training experience. All residents completedthe ATSS and the 27 items of the original Blumenthalet al. questionnaire (7).

RESULTS

Internal consistency was evaluated by Cronbach’s al-pha. For the 16 items on the new scale, � � 0.97. Themean interitem correlation was 0.69. If the generalsubstance abuse and delirium items from the Blu-menthal et al. survey (7) were included in the anal-ysis, Cronbach’s alpha remained at 0.97, and themean interitem correlation was 0.67. These values areconsistent with the hypothesis that the new question-naire would show appropriate internal consistency.

Dimensionality of the ATS was evaluated usingfactor analysis, which is a method that examines acorrelation matrix among a set of variables to deter-mine whether one or more variables can be con-structed to represent the intercorrelations among theoriginal variables. It is then possible, using factorloadings, to examine the correlation of each variablein the original data set with these constructed vari-ables, the factors, and to interpret what each factormeans and what the original variables have in com-mon that produces a particular factor. Principal com-ponents factoring of the 16 items yielded two factors,with Eigenvalues exceeding 1.00 and accounting for77.5% of the total variance of the 16 items. Factorloadings, after application of a normalized varimaxrotation, are presented in Table 1. Only the highestloadings of each factor were used for interpretation.While some factor analysts will interpret loadingsdown to 0.35 or even 0.30, this does not seem reason-able in a pilot study that factors data from such asmall sample. The highest loading items on the firstfactor related to confidence in withdrawing patientsfrom addictive substances. Items reflecting familiar-ity with nonmedical approaches to treatment loadedmost highly on the second factor (i.e. psychothera-pies, structured living programs, and 12-step meet-ings). We entitled the first factor “Medical Ap-

SATTAR ET AL.

Academic Psychiatry, 28:3, Fall 2004 http://ap.psychiatryonline.org 207

TABLE 1. Varimax Normalized Factor Loadings

Factor 1 (a) Factor 2 (b)

Withdrawing patients from alcohol .887689* .228352Withdrawing patients from benzodiazepines .865931* .289617Withdrawing patients from cocaine .846008* .367549Psychotherapies for addiction patients .179361 .925400*Structured living programs .289825 .850832*Understanding of 12-step meetings .413244 .785498*Understanding of alternatives to 12-step programs .511824 .645927Alternative methods to preventing relapse .518763 .678513Pharmacotherapies for preventing relapse .585151 .537936Provide therapy for patients with active substance abuse .606053 .683365Withdrawing patients from barbiturates .637405 .274258Diagnose and treat ‘‘dual diagnoses’’ patients .674051 .605669Withdrawing patients from opiates .729513 .427881Working with patients who are actively abusing substances .757051 .433874Treating patients .779392 .542665Withdrawing patients from stimulants .840143* .347674

* Highest factor loadings on each factor(a) Medical Approaches To Addiction Treatment(b) Non-Medical Approaches To Addiction Treatment

proaches to Addiction Treatment,” while the secondfactor was entitled “Non-Medical Approaches to Ad-diction Treatment.”

Multivariate analysis of variance (MANOVA)was conducted using factor scores measuring the twodimensions of the new questionnaire. First, MAN-OVA showed a significant effect for years of resi-dency, F (8, 30) � 3.60, p � 0.004. Univariate valuesindicated that the differences among years of resi-dency were significant only for “Medical Approachesto Addiction Treatment Factor,” F (4, 16) � 4.13, p �

0.017, with more advanced residents showing higherscores. Second, MANOVA showed a significant effectof participating in a specific training program for sub-stance abuse, F (2, 16) � 6.94, p � 0.005. Univariateresults indicated a significant difference for “Non-Medical Approaches to Addiction Treatment Factor,”F (1, 19) � 10.98, p � 0.004, with residents who hadcompleted a specific substance abuse rotation achiev-ing higher scores.

DISCUSSION

Our results suggest that the ATS is related to the con-fidence and preparedness that residents express intheir ability to treat substance abuse problems. Ad-ditionally, the ATS appears to represent two factors,“Medical Approaches to Addiction Treatment Fac-tor,” reflecting medical knowledge, and “Non-Medi-

cal Approaches To Addiction Treatment Factor,” re-flecting psychosocial interventions. Our resultsindicate adequate internal consistency and suggestthat general medical knowledge and psychosocialknowledge, as measured by the two factors, are in-dependent components of substance abuse treatmentconfidence. The validity of the questionnaire and theinterpretation of the two factors are supported by theassociation between 1) the “Medical Approaches toAddiction Treatment Factor” and years of psychiatryresidency training and 2) the “Non-Medical Ap-proaches To Addiction Treatment Factor” and specificsubstance abuse training. Thus, the “Medical Ap-proaches to Addiction Training Factor” appears to beheavily influenced by the extent of training that res-idents have had within the discipline of psychiatry,while the “Non-Medical Approaches to AddictionTreatment Factor” is most highly correlated with spe-cific exposure to substance abuse training.

We recognize that the generalizability of these re-sults is limited by the small sample size. The presentstudy needs to be replicated with a larger number ofsubjects and in different areas of the country. The con-clusion regarding the factor structure of the ATS isclearly preliminary. The small number of cases in thispilot study is inadequate to support assumptions ofa stable factor structure and there is strong cross load-ing of items on the factors derived in this study. Fac-tor methods used in the preset study are relatively

ADDICTION TRAINING SCALE PILOT STUDY

208 http://ap.psychiatryonline.org Academic Psychiatry, 28:3, Fall 2004

simple and direct, chosen out of recognition of thelimitations imposed by our small sample size. Rep-lication with a larger sample size will allow applica-tion of more complex factor analytic techniques, in-cluding examination of the possibility that the factorsidentified may be correlated. We note that some itemsthat intuitively belong with Factor I, medical ap-proaches, do not show a clear association with thatfactor when compared with Factor II. It is possiblethat a larger sample size may also clarify the relation-ship of specific items to the factors and improve factorstability. Additionally, future studies should correlatethe ATS with the individual subject’s PRITE resultsto further validate this questionnaire. Unfortunately,since participants completed the questionnaires forthis study anonymously, there was no mechanism formatching PRITE and ATS scores.

Despite these limitations, initial results are prom-ising and somewhat remarkable in their strength, inview of the small sample size. The results suggest thatusing the ATS may be of benefit in evaluating resi-dents’ self-reported preparedness to treat substanceabuse disorders.

Even though assessing residents’ competence intreating clinical disorders should not be limited tosimply asking them about their confidence (7), self-report scales such as the ATS can provide valuablefeedback about residents’ view of their training. Someresidents will inaccurately estimate their abilities totreat the disorders listed on the ATS, but even thesedata can provide important information to the train-ing directors, especially if residents identify weakness

in certain areas of substance abuse training (8). Wepropose that the ATS be used to complement otherstrategies for evaluating the competency of residentsin substance abuse treatment and not as a substitutefor clinical supervision and appropriate examinationof their knowledge base. Completing the ATS at thebeginning of the rotation can identify areas where res-idents perceive themselves to be unprepared. Thiswill help faculty and residents focus on specific areasof substance abuse training. Completing the ATS atthe end of the rotation will measure the progress res-idents think they have made in addressing their per-ceived deficits in preparedness to treat substanceabuse disorders. For residents to continue to identifyareas of weakness suggests that extra work is neededto address deficiencies.

CONCLUSION

Our results show that the ATS could be beneficial inevaluating psychiatry residents’ substance abusetraining. To verify these findings, this study shouldbe replicated with a larger sample size. If supportedby future research, the ATS could be a useful self-report evaluation tool for psychiatry residents’ sub-stance abuse training. We would like to reinforce thatthe ATS is not a substitute for traditional methods ofresident evaluation. However, using the ATS may aidfaculty in developing an individualized addictionpsychiatry training program for residents and pro-vide a measure of progress toward addressing iden-tified shortcomings.

References

1. Anda R, Williamson D, Remington P. Alcohol and fatal in-juries among US adults: finding from the NHANES 1 epi-demiologic follow-up study. JAMA. 1988. 260:2529–2532

2. Rice D. The Economic Cost of Alcohol Abuse and AlcoholDependence: 1990. Washington, DC: National Institute on Al-cohol Abuse and Alcoholism, National Institutes of Health.1993:10–11

3. Eckardt M, Hartford T, Kaelber C. Health hazards associatedwith alcohol consumption. JAMA. 1981. 246:648–661

4. Baker FM, Haning WF. Substance abuse and dependence ina public hospital. Hawaiian Medical Journal. 2001. 60:35–38.

5. Brems C, Johnson ME, Bowers L, Lauver B, Mongeau VA:Comorbidity training needs at a state psychiatric hospital.Adm Policy Ment.Health 2002; 30:109–120

6. Blumenthal D, Gokhale M, Campbell EG, et al.: Preparednessfor clinical practice: reports of graduating residents at aca-demic health centers. JAMA. 2001. 286(21):1027–1034

7. Stimmel B, Cohen D, Colliver J, et al.: An Assessment ofHouse Staff’s Knowledge of Alcohol and Substance AbuseUtilizing Standardized Patients. Subst Abus 2000; 21:1

8. Greenberg WM, Ritvo JI, Fazzio L, et al.: A survey of addic-tion training programming in psychiatry residencies. Aca-demic Psychiatry. 2002. 26:105–109.

9. Rothschild BM. Graduating residents’ perception of theirpreparedness for practice. JAMA. 2001. 286:2666.

10. Blumenthal D. Graduating residents’ perception of prepared-ness for practice. JAMA. 2001. 286:2667.