The Arts in Psychotherapy, Vol. 20, pp. 8S88, 1993 Printed in the USA. All rights reserved.

0197-4556/93 $6.00 + .OO Copyright 0 1993 Pergamon Press Ltd.

RELIABILITY AND VALIDITY TESTS OF THE DIAGNOSTIC

DRAWING SERIES

ANNE MILLS, MA, ATR, BARRY M. COHEN, MA, ATR and JACKIE Z. MENESES, BA*

Almost half a decade ago, this journal introduced a new art therapy assessment, the Diagnostic Drawing Series (DDS) (Johnson, 1988). This instrument was created because of the clinical imprecision of existing art therapy assessment procedures and the absence of a research foundation on which to base clinical judg- ments (Cohen, Hammer, & Singer, 1988). Since that time, development of the DDS has continued, with updates being shared through limited circulation newsletters or professional presentations.

The Diagnostic Drawing Series, designed by art therapists, is an art interview in which collection and picture analysis is linked to DSM-III and DSM-III-R diagnoses (APA, 1980, 1987). The protocol for the three-drawing assessment tool incorporates instruc- tions for unstructured, structured and semi-structured drawing tasks (Benjamin, Hopkins, & Nation, 1987; Cohen, 1990; Turkington, 1985; Wade & Tavris, 1987).

Presented with three pieces of 18 by 24 inch white paper and a 12-color box of square, soft chalk pastels, the client is asked initially to “Make a picture using these materials. ” The instruction for the second piece of paper is “Draw a picture of a tree. ” The third request is “Make a picture of how you are feeling, using lines, shapes, and colors” (Cohen, 1985).

This article will review numerous reliability and validity tests of the DDS. In so doing, it will have the effect of doubling the total number of such tests pub- lished in the entire art therapy literature. Reliability and validity tests have been reported infrequently re- garding other art therapy assessments (Elin & Nucho, 1979; Ferguson & Debevec, 1990; Hiltunen, 1990; Levick, 1983; McGlashan, Wadeson, Carpenter, & Levy, 1977; McIntyre & Wright, 1979; Rubin, Schachter & Ragins, 1983; Silver, 1990). Because documenting reliability and validity is “essential” and of “overreaching importance” (APA, 1974, p. 16), such tests were among the earliest work with the DDS. Where an assessment is presented without such substantiation, both clinician and researcher must ask whether the instrument and results have value and are worthy of acceptance and further exploration (Lyerly, no date).

Validity Tests of the DDS

The procedures that have been used to validate the DDS will be discussed according to the format sug- gested by MacFarlane (1942). Validity in the strictest sense, that is, adequacy of the tool for the task, is not discussed here. It is not testable in psychological re-

*Anne Mills, an art therapist in private practice with Art Therapy Services of Alexandria, VA, also works for the George Washington

University Counseling Center and is the Director of the Diagnostic Drawing Series archive.

Barry Cohen, Program Director of the Center for Abuse Recovery & Empowerment, located at the Psychiatric Institute of Washington, DC, is the first editor of Multiple Personality Disorder From the Inside Out, a book of first-person accounts about living with MPD.

Jackie Meneses, who worked extensively with the DDS project in 1984-86, is now employed by a residential substance abuse program for

adolescents in California. The authors gratefully acknowledge the assistance of Adrienne Kwapien, MSW, ATR, in data collection, Jeffrey Hammer, PhD in data

analysis and presentation, and the encouragement of Thomas Wise, MD, Chairman, Department of Psychiatry, Fairfax Hospital.

83

84 MILLS, COHEN AND MENESES

search that lacks an absolute standard (i.e., for “san- ity”) against which to compare. At best, a “proce- dural validity” is testable (Mills, 1989). An example of this, as seen below, would be multiple compari- sons of a new diagnostic interview with known instruments.

As it is of critical importance to standardize pro- tocol for the administration of an assessment, the DDS Handbook specifies the art materials to be used, what instructions are to be given the participants and how much time is allowed for each task (Cohen, 1985). To check that research using the DDS is done systematically and according to the protocol described in the Handbook, DDSs are organized into one ar- chive and their collection is coordinated from this point. The archive of DDSs also serves as a central clearinghouse to gather and disseminate information regarding the technique. Administration of the assess- ment is done primarily by registered art therapists, who possess similar training (Cohen, 1990).

The general psychiatric inpatient population was the primary focus. The archive includes virtually all DSM-III-R psychiatric disorders and some medical populations. As research should utilize a large sample with a control population, the DDS archive, which contains over 1000 cases, includes a control sample. The DDSs were gathered through collaborative work by art therapists in institutions that were broadly dis- tributed across the United States. For clarity and to attempt to control for the effects of medications, it is specified that Series may be collected for research no more than five days after admission to hospital. It should be noted that the DDS is the only art therapy assessment to incorporate all of the above aspects of sound research design.

Testing correlations with selected criteria, such as comparing the artwork of many subjects from one diagnostic group, yields normative data. In fact, much of the DDS work to date is in the form of normative studies, although many investigations are not yet published. In one approach, researchers may look at the graphic profile of subjects grouped by

diagnoses assigned by psychiatrists and psycholo- gists. Norms have been established for diagnostic groups that include controls, and individuals diag- nosed with major depression, dysthymia and schizo- phrenia (Cohen, Hammer, & Singer, 1988), border- line personality disorder (Mills, 1989), multiple per- sonality disorder (Mills & Cohen, 1993) and organic mental syndrome (Couch, 1992).

Non-psychiatric groupings (e.g., Sobol & Cox on sexually abused children, 1992), medical (e.g., Cheyne-King’s work on head injury, in Cohen, Al- berts, Cheyne-King, Tripp, & Mills, 1986) and cross- cultural studies (e.g., Yamashita’s work on the Jap- anese, in Cohen, Mills, Cox, Leavitt, & Yamashita, 1989) have also been explored. Attesting to the va- lidity and usefulness of these normative studies were the outcomes from one of the first replication studies in the field of art therapy. Many of the same results were found between two studies of DDSs drawn by two different samples of individuals diagnosed with multiple personality disorder (Kress, 1992).

Comparison with another test to check for validity by consistency in response was done in 1986 with a single subject. In this experiment, a male psychiatric inpatient participated in a number of projective draw- ing sessions and art interviews. The subject was from a dysfunctional family, a heterosexual transvestite who found his own impulse to cross-dress to be in- creasingly upsetting. The first two authors (A.M. and B .M.C .) conducted individual and group art therapy sessions with this individual in two different hospitals across two months, which spanned three psychiatric admissions. The tasks administered individually were as follows: a DDS upon admission; a House-Tree-Person (Buck, 1948)‘; Draw-a-Person-of-the-Opposite-Sex (Levy, 1958)‘; an Ulman Personality Assessment Pro- cedure (Ulman, 1975)3; a second admission DDS; and a DDS upon discharge from the third hospitalization.

Mills (1989) noted that the different tools yielded complementary, not conflicting, information. In this trial the DDS elicited consistent responses that are comparable to the other expressive and projective

‘A number 2 pencil with eraser, white paper, 7” X 8%“, presented horizontally or vertically depending on the drawing number. “Draw as

good a house as you can”, etc.

*“Moderately soft” pencil and 8%” X 11” white paper. “You drew a__ figure” or “This is L figure,” “Now please draw a” (p.

88).

‘Twelve colors of hard pastels, a full box of each color, an easel, four pieces of 18” X 24” grey construction paper. Drawings: (1) “Please

use these materials to make a picture”; (2) record of prescribed physical movements; (3) scribble that is developed into a picture; and (4) free

choice picture.

TESTS OF DIAGNOSTIC DRAWING SERIES 85

tools used, but each test also made a unique contri- bution. The similarities of the DDS to the other draw- ing assessments were moderately strong, but not enough to constitute a duplication. Important similar- ities in the DDSs in test-retest situations were noted. The advantages of using the DDS rather than the other techniques tested are its ease of administration com- bined with media that foster self-expression.

The same validity test, comparison of one test with a better known one, was performed with a group of subjects as well. In this case, comparison of DDSs with a projective drawing task and self-reports showed little correspondence between the DDS and any of the other procedures (Leavitt, 1988). The DDS was compared with the Depression Self Rating Scale (Birleson, 1981), child and parent versions of the Children’s Depression Inventory (Kovacs & Beck, 1977), and the Draw-a-Person (Machover, 1949). The association between DDSs and Depression Self Rating Scales was very weak. No ~lationship was found between the DDSs and the depression indica- tors of the Draw-a-Person (nor were those Draw-a- Person items found related to depression in the child subjects). Leavitt noted an art therapist with an 80% capacity to accurately disc~minate depression in drawings used some criteria similar to those looked for in DDS rating “but different enough so that they were not noted by the DDS” (p. 126). It should be pointed out that in this study the researcher altered the materials (smaller paper), the rating system and the age range of subjects (children) from the standard DDS protocol. In a sense, this functions as a discrim- inant validation, in that this version of the DDS does not correlate significantly with variables from which it should differ, by virtue of its m~i~~ation.

Validity can also be tested by comparison of pro- jective material with life history material. A test that is related involves asking one or more experts to make blind diagnostic judgments, as in Leavitt’s study above. Similarly, Wi~lin and Augusthy (1988) re- ported an art therapist’s ability to accurately discrim- inate five diagnostic categories to 77.3% accuracy. However, these inquiries examine clinical acumen or predictors in the art rather than the tool itself.

Degree of success in prediction has not been fully tested to investigate validity of the DDS. However, a trial featuring a small sample of DDSs was examined by a prototype of artificial intelligence programmed with DDS results. The program was able to predict degree of belongingness in four possible diagnostic

categories with a 77% accuracy rate (Cohen & Anon- ymous, 1987).

These trials suggest high validity with reference to the particular use for which the test is being considered.

Reliability Tests of the DDS

The use of a rating scale is a cornerstone of DDS research, but need not be used by a clinician. Rating is a way to separate the elements of a picture into independent variables that may be tested (Wadeson, 1978) and is a springboard to testing validity and reliability. Rating is defined here as requiring opera- tional de~nitions of specific criteria (Elin & Nucho, 1979; Kwiatkowska, 1978; Reiner, Tellin, 8r O’Reilly, 1977; Wadlington & McWhinnie, 1973; Zierer & Zierer, 1960). Judging is defined as involv- ing global responses that are relatively more subjec- tive and less testable (Horowitz, 1970; Rubin & Schachter, 1972; Ulman & Levy, 1975).

Rating the DDS involves 23 categories, each de- scribed as objectively as possible in the DDS Rating Guide (Cohen, 1986). Some rating criteria offer more than two options. That is, some categories force yes/ no, present/not present decisions whereas others offer several optional descriptors that may be chosen. A total of 183 decisions per Series are demanded of a rater. The development of the DDS Rating Guide was shaped by existing rating systems, by the thoughts of a number of art therapists and by theoretical writings on the structure of art (Dondis, 1973; Feldman, 1972; Kreitler & Kreitler, 1972).

The most common test of reliability is that of con- curring rating judgments among different raters. After a training period of two months involving revision of the Rating Guide, two of the authors (J.Z.M. and B.M.C.) each rated 30 Series to determine interrater reliability. The results were analyzed by the Fist au- thor and are presented in Table 1. The overall per- centage of agreement between these two experienced raters, one a non-art therapist, was 95.7%. According to Drummond (1983), this fulfills the usual and cus- tomary level of interrater reliability. It exceeds many comparable studies in the field and suggests a very reliable rating system.

Table 1 contains a condensed version of the DDS Rating Guide. Interested readers are referred to the Guide itself for full definitions of terms (Cohen, 1986). Criteria that are examples of low and high

MILLS, COHEN AND MENESES

Table 1

Interrater Reliability Study

Category/No. Choices Percent Agreement

Color types (3) 98

Blending (2) 92

Idiosyncratic color (2) 97

Line/shape mix (3) 98

Integration (3) 94

Abstraction (3) 88

Representation (3) 77

Image (3) 99

Enclosure (2) 98

Groundline (2) 100

People (2) 100

Animals (2) 99

Inanimate objects (2) 97

Abstract symbols (2) 93

Word inclusion (3) 100

Landscape/water (3) 93

Line Quality/pressure (3) 97

Line length (3) 99

Movement (3) 97

Space usage (4) 92

Tree (4) 96

Tilt (2) 99

Unusual placement (2) 99

Overall average percentage of agreement 95.7

An example of high reliability is the criterion called groundline, which achieved interrater reliabil- ity of 100%. According to the rating guide, ground- line is an element found in representational pictures only. It is rated as either present or absent and is defined as a surface other than the page edge. In a tree picture, a groundline is defined as lines other than roots that extend at least one inch on both sides of the trunk.

reliability are defined here in a simplified form for the reader’s benefit.

Table 1 shows abstraction and representation achieved relatively low reliability. They are complex to define and the procedure to rate them is more com- plex than present/not present. They are mutually ex- clusive categories. Only one, abstraction or represen- tation, is rateable per picture, so the rater must re- member to make the choice and leave the other blank. Representational pictures are those with easily iden- tifiable images, whereas abstractions are defined as those where the image is not easily recognizable. Having established a picture as one or the other for

rating purposes, the rater determines whether it is predominantly geometric or biomorphic, angular or curvilinear (terms used within abstraction and repre- sentation, respectively) or a mixture with neither pre- dominating. Interrater agreement among these six possible choices (three choices for each of two cate- gories) was 88% (abstraction) and 77% (representation).

A paper comparing the DDSs of child controls and those of children with a diagnosed psychological dis- order investigated the interrater reliability between the second author, a trained rater, and an untrained rater. High reliability was found on six of the seven rating categories that significantly discriminated between the artwork of the control and treatment groups (Neale, 1991).

A 1986 concordance study of 29 naive raters (art therapists untrained in the DDS rating system and using it for the first time), analyzing a single DDS, yielded an average of 77% correct ratings across all criteria (Mills, 1989). The test was performed under difficult circumstances, and a number of participants left before completing the rating. Blank or incomplete responses were scored as “incorrect.” Therefore, as high as the percentage of agreement was, it may be misleadingly low and not accurately reflect partici- pants’ actual comprehension of the Rating Guide.

These tests suggest that this tool possesses high reliability with reference to these applications.

Conclusion

This paper details the reliability and validity tests of the Diagnostic Drawing Series (DDS). Aspects of the DDSs design that contribute to its validity and reliability are reviewed. Five investigations of the va- lidity of the DDS are described. Three interrater re- liability tests are reported. Thus, evidence accrues that the Diagnostic Drawing Series is a reliable and valid instrument for art therapists to use in clinical assessment and research.

The authors invite colleagues who use assessments to consider the DDS, which is carefully designed and supported by systematic research. Rather than con- tributing to the proliferation of art therapy assess- ments, each of which cannot be compared to the next because of their differences, art therapists may prefer a tool that offers the possibility of benefitting from ongoing research (Mills & Goodwin, 1991). The DDS may be used clinically with a variety of thera- peutic approaches, yet it also holds the potential to contribute to a growing central body of art therapy

TESTS OF DIAGNOSTIC DRAWING SERIES

knowledge. It is well suited to collaborative research and replication, in that standardization of administra- tion allows sharing between researchers who are geo- graphically or philosophically distanced. It is hoped that both clinicians and researchers, when they are reminded of the old motto of the Tavistock Institute in London, “No therapy without research; no research without therapy,” will agree that the two need not be mutually exclusive.

References

American Psychiatric Association. (1980). Diagnostic and statis- tical manual of mental disorders (3rd ed.). Washington, DC: Author.

American Psychiatric Association. (1987). Diagnostic and statis- tical manual of mental disorders (rev. 3rd ed.). Washington, DC: Author.

American Psychological Association. (1974). Frederick B. Davis, Chair, Joint Committee. Standardsfor educational andpsycho- logical tests. Washington, DC: Author.

Benjamin, L. T., Hopkins, J. R., & Nation, J. R. (1987). Psy- chology. New York: Macmillan.

Birleson, P. (1981). The validity of depressive disorder in child- hood and the development of a self-rating scale: A research report. Journal of Child Psychology andPsychiatry, 22, 73-88.

Buck, J. N. (1948). The H-T-P test. Journal of Clinical Psychol- ogy, 4, 151-159.

Cohen, B. M. (Ed.). (1985). The Diagnostic Drawing Series Handbook. (Available from Barry M. Cohen, P.O. Box 9853, Alexandria, VA 22304).

Cohen, B. M. (Ed.). (1986). The Diagnostic Drawing Series Rat- ing Guide. (Available from Barry M. Cohen, P.O. Box 9853, Alexandria, VA 22304).

Cohen, B. M. (1990). Diagnostic Drawing Series. In I. Jakab, (Ed.), Stress management through art: Proceedings of the In- ternal Congress of Psychopathology of Expression. (pp. 123- 130). Boston, MA: ASPE.

Cohen, B. M., Albcrts, L. A., Cheyne-King, S., Tripp, T., & Mills, A. (Speakers). (1986). The Diagnostic Drawing Series: Research update (Cassette Recording No. S303-6). Los Ange- les, CA: American Art Therapy Association.

Cohen, B. M., & Anonymous. (1987). Wiz. Unpublished com- puter program.

Cohen, B. M., Hammer, J. S., & Singer, S. (1988). The Diag- nostic Drawing Series: A systematic approach to art therapy evaluation and research. The Arts in Psychorherapy, 15 (l), 11-21.

Cohen, B. M., Mills, A., Cox, C. T., Leavitt, C. G., & Yama- shita, Y. (1989, November). The Diagnostic Drawing Series: Its use in clinical practice. One-day workshop presented at the annual national meeting of the American Art Therapy Associ- ation, San Francisco, CA.

Couch, J. B. (Speaker). (1992). Organic mental syndrome; Diag- nostic Drawing Series research with the elderly (Cassette Re- cording No. 16). Las Vegas, NV: American Art Therapy As- sociation.

Dondis, D. A. (1973). A primer of visual literacy. Cambridge, MA: The MIT Press.

Drummond, T. B. (1983). They could not talk, so they drew [Re- view]. American Journal of Art Therapy, 22, 69-71.

Elm, N., & Nucho, A. 0. (1979). The use of Kinetic Family Drawing as a diagnostic tool in assessing the child’s self- concept. Art Psychotherapy, 6 (4). 241-248.

Feldman, E. B. (1972). Varieties of visual experience. New York: Harry N. Abrams.

Ferguson, W., & Debevec, D. (1990). Screening kindergarten stu- dents: An art therapy assessment. Art Therapy, 7 (3). 119-125.

Hiltunen, S. S. (1990). The effects of art/drama therapy experi- ences on rigidity, body concept and mental maturity in graphic thinking of adolescents with mental retardation. Japanese Bul- letin of Arts Therapy, 21, (I), 165-175.

Horowitz, M. J. (1970). Image formation and cognition. New York: Appleton-Century-Crofts.

Johnson, D. R. (Ed.). (1988). Assessment in the creative arts ther- apies [Special issue]. The Arts in Psychotherapy, 15, (1).

Kovacs, M., & Beck, A. T. (1977). An empirical-clinical ap- proach toward a definition of childhood depression. In J. G. Schulterbrandt & A. Allen, (Eds.), Depression in childhood: Diagnosis, treatment and conceptual models (pp. l-25). New York: Raven Press.

Kreitler, H., & Kreitler, S. (1972). Psychology of the arts. Durham, NC: Duke University Press.

Kress, T. (Speaker). (1992). The Diagnostic Drawing Series and multiple personality disorder: A validarion study (Cassette Re- cording No. 55). Las Vegas, NV: American Art Therapy As- sociation.

Kwiatkowska, H. Y. (1978). Family therapy and evaluation through art. Springfield, IL: Charles C Thomas.

Leavitt, C. G. (1988). A validity study of the Diagnostic Drawing Series as used for assessing depression in children and adoles- cents. Unpublished doctoral dissertation, California School of Professional Psychology, Los Angeles.

Levick, M. F. (1983). They could not talk and so they drew: Chil- dren’s styles of coping and thinking. Springfield, IL: Charles C Thomas.

Levy, S. (1958). Projective figure drawing. In E. F. Hammer (Ed.), The clinical application of projective drawings (pp. 83- 112). Springfield, IL: Charles C Thomas.

Lyerly, S. B. Handbook of Psychiatric Rating Scales. National Institute of Mental Health, U.S. Department of Health, Edu- cation, and Welfare. No date.

MacFarlane, J. W. (1942). Problems of validation inherent in pro- jective methods. American Journal of Orthopsychiatry, 12, 405-410.

Machover, K. (1949). Personality projection in the drawing of the human figure. Springfield, IL: Charles C Thomas.

McGlashan, T. H., Wadeson, H. S., Carpenter, W. T., & Levy, S. T. (1977). Art and recovery style from psychosis. The Jour- nal of Nervous and Mental Disease, 164 (3). 182-189.

McIntyre, P., & Wright, J. (1979). Family Drawing Depression Scale. Proceedings of the Tenth Annual Conference of the American Art Therapy Association. Washington, DC: AATA Publications Committee.

Mills, A. (1989). A statistical study of the formal aspecrs of the Diagnostic Drawing Series of borderline personality disor- dered patients, and its context in contemporary art therapy. Unpublished master’s thesis, Concordia University, Montreal, Canada.

MILLS, COHEN AND MENESES

Mills, A., & Goodwin, R. (1991). A survey of assessment use in child art therapy. Arr Therapy, 8 (2), 10-14.

Mills, A., & Cohen, B. M. (1993). Facilitating the identification of multiple personality disorder through art: The Diagnostic Drawing Series. In E. Kluft (Ed.). Expressive and fun&ma1 therapies in the treatment of multiple personality disorder. Springfield, IL: Charles C Thomas.

Neal& El L. (1991). The Children’s Diagnostic Drawing Series. Unpublished master’s thesis, Universitv of Louisville. KY.

Reiner, E. R., Tellin, J., & O’Reilly, J. (1977). A picture regres- sion scale for adults. Art Psychotherapy, 4, 21%223.

Rubin, J. A., & Schachter, .I. (1972). Judgments of psychopathol- ogy from art productions of children. Confinia psychiatrica, IS, 237-252.

Rubin, J. A., Schachter, J., & Ragins, N. (1983). Intraindividual variability in human figure drawings: A developmental study. American Journal of Orthopsychiatry, 53 (4), 654667.

Silver, R. A. (1990). Silver drawing test of cognitive skills and adjustment. New York: Ablin Press.

Sobol, B., & Cox, C. T. (Speakers). (1992). Art and childhood dissociation: Research with sexually abused children (Cassette Recording No. 59). Las Vegas, NV: American Art Therapy Association.

Turkington, C. (1985). Therapist seeks correlation between diag- nosis, drawings. APA Monitor, 26 (4). 34-36.

Ulman, E. (1975). A new use of art in psychiatric diagnosis. In E. Ulman & P. Dachinger, (Eds.), Art therapy in theory andprac- rice (pp. 361-386). New York: Schocken Books.

Ulman, E., & Levy, B. I. (1975). An experimental approach to the judgement of psychopathology from paintings. In E. Ulman & P. Dachinger, Art therapy in theory and practice (pp. 393- 402). New York: Schocken Books.

Wade, C., & Tavris, C. (1987). Psychology. New York: Harper & Row.

Wadeson, H. (1978). Some uses of art therapy data in research. American Journal of Art Therapy, 18 (I), 11-18.

Wadlington, W. L., & McWhinnie, H. J. (1973). The develop- ment of a rating scale for the study of formal aesthetic qualities in the paintings of mental patients. Art Psychotherapy, I, 201- 220.

Wittlin, B. W., & Augusthy, R. (1988, December). Comparison of art psychotherapy and discharge diagnoses of diagnostic unit patients. Art Therapy, 94-98.

Zierer, E., & Zierer, E. (1960, July). The integration component in creative therapy. Journal of the Hillside Hospital, 156-170.