Computers in Human Behavior, Vol. 2, pp. 111-116, 1986 0747-5632/86 $3.00 + .00 Printed in the U.S.A. All rights reserved. Copyright © 1986 Pergamon Journals Inc.

Paper-and-Pencil Versus Computer-Administered MMPIs:

A Comparison of Patients' Attitudes

Ronald H. Rozensky, Laurie Feldman Honor, 1 Kenneth Rasinski, 2 Steven M. Tovian

and Gordon I. Herz 3

Department of Psychiatry, Evanston Hospital, Evanston, Illinois

and

Northwestern University Medical School

Abstract--Attitudinal variables contributing to patients'perceptions of computer-administered MMPIs were examined in a heterogeneous clinical sample. One hundred and eighty~ve subjects were randomly assigned to computer or paper-and-pencil administration conditions in which they received both the M M P I and an attitude survey. In general, the computer administered M M P I was perceived as more interesting and less anxiety-provoking than the paper-and-pencil MMPI . Subjects in the computer group felt more positively toward the examiner administering test instructions and felt that the test went more quickly. Subjects in the computer and paper-and-pencil groups did not differ regarding their concerns about privacy and the disposition of test results. Results suggest that cyberphobia does not characterize the general clinical population and that patients' attitudinal and affective responses to computerized versions of the M M P [ may enhance the utility of this assessment procedure.

Over the past several years the use of the computer to administer, score and interpret the Minnesota Muhiphasic Personality Inventory (MMPI) has gained increasing popularity and acceptance in a wide variety of clinical settings (Adams & Shore, 1976; Finney, 1966; Fowler, 1969; Lachar, 1974a). As computer tech- nology develops, it becomes clear that automated M M P I assessment provides a cost-effective method to increase the efficiency of mental health professionals (Byers, 1981; Elwood, 1969; Evan & Miller, 1969; Finney, 1966; Lushene, O'Neil, & Dunn, 1974; Smith, 1963).

The authors wish to thank Ms. Meg Holland for her assistance in data collection

Requests for reprints should be sent to Ronald H. Rozensky, Department of Psychiatry, Evanston Hospital, 2650 Ridge Avenue, Evanston, IL 60201.

1Now at University of Colorado Health Sciences Center. 2Now at University of Chicago. 3Now at Memphis State University.

111

112 Rozensky et al.

While research has demonstrated that conclusions generated from computerized MMPIs typically are as reliable and valid as traditional M M P I methods (Adams & Shore, 1976; Elwood, 1972a, 1972b; Katz & Dalby, 1981; Lushene et al., 1974; Orr, 1969), it has been continually suggested that patients' attitudes and reactions to computerized test administration may have an adverse impact on the validity of the data (Evan & Miller, 1969; Hedl, O'Neil & Hansen, 1973; Slack & Van Cura, 1968). Duthie (1984), for example, argues that the mystique associated with computers may, in fact, establish a response set that could affect the test-taking attitudes of computer test respondents. Likewise, Hofer and Green (1985) "sug- gest that the psychologist must be aware of the effect of computerized test admin- istration on the test taker and not assume everyone is comfortable with the machine" (p. 829). Recent surveys of mental health practitioners' attitudes towards computers reveals that many still feel that "human services and medical care have no rela- tion to computers" (Pinkerton & Raffoul, 1984, p. 64) and that the "use of a com- puter implies they lack humanity" (Schwartz, 1984, p. 67).

Clearly, despite the growing popularity of many computerized tests such as the M M P I and the accompanying supportive empirical evidence regarding reliabil- ity and validity, many professionals remain very uncomfortable with these proce- dures. Although it is not yet possible to identify all of the sources contributing to this discomfort, it is thought that concerns related to patients' attitudinal and affec- tive responses to computerized assessments have been a significant factor. Unlike the psychometric properties of computerized tests, these issues have yet to receive intensive empirical scrutiny. The purpose of this study was to investigate the attitu- dinal and affective reactions towards computerized versus paper-and-pencil MMPIs among a heterogeneous clinical population.

METHOD

Subjects

Subjects were 185 consecutive patients referred to the Psychodiagnostic Evalua- tion and Testing Service of Evanston Hospital, Department of Psychiatry. Patients were hospitalized for psychiatric problems or for alcoholism treatment, in outpa- tient psychotherapy, or in psychiatric day treatment. The M M P ! was either the only test or the first test administered in a battery. There were 134 men and 51 women with a mean age of 35.06 years ( s d = 13.3).

Procedure

Subjects were randomly assigned to take either the computer or paper-and-pencil version of the MMPI . Prior to beginning the test, subjects were oriented to the task by an administrative aide who explained the general nature of the procedure. This aide was available to assist subjects during the test, as needed. All subjects in the paper-and-pencil group worked on the M M P I (Form-R) at their own pace either in their hospital room or another quiet well-lighted test room provided for the examination. Subjects in the computer group were seated in front of a com- puter terminal consisting of a display monitor and keyboard. The terminal was linked to an AlphaMicro computer which had been programmed to present the 566 MMPI items and record and store the subjects' responses. Identical directions

Patient attitudes towards the computerized M M P I 113

appeared on the screen and in the F o r m - R booklet. Test items appeared on the screen one at a time. After the subject responded by typing in " T " or "F," the next item appeared on the screen. Responses from subjects in the computer group were transferred to F o r m - R answer sheets and hand scored along with the responses from the paper-and-pencil group. A total of thirteen profiles were eliminated as invalid utilizing validity rules similar to those of Lachar (1974b). The remaining sample included 86 computer subjects (60 men and 26 women) and 86 paper-and- pencil subjects (64 men and 22 women).

After finishing the M M P I , each subject completed an attitude survey about the test they had just taken. Subjects in the paper-and-pencil group did so using paper and pencil while those in the computer group completed this attitude survey using the computer.

Attitude Survey

A Computer Attitude Survey (CAS) was constructed for the purposes of this study. It consisted of 30 Likert-type statements scaled from 1 to 4 (1 --strongly agree, 2 = somewhat agree, 3 = somewhat disagree, and 4 = strongly disagree). Groups of attitudinal items were designed to measure a range of responses to psychologi- cal testing that were either noted clinically by the authors or cited in the litera- ture (Cogswell and Estovan, 1965; Evan & Miller, 1969; Hedl, et al., 1973; Lushene, et al., 1974; Slack, Hicks, Reed, & VanCura, 1966; Smith, 1968). Item presentation was counter-balanced to prevent response bias.

The CAS contained eight scales. The scale names and brief descriptions follow:

(a) Affect--how relaxed or comfortable the patients felt while taking the MMPI . (b) C o g n i t i v e - h o w positive, accurate and interesting the patients thought the

test-taking experience to be. (c) T ime- -how quickly the time seemed to pass during the evaluation and how

efficiently time was used. (d) Disposi t ion--how concerned patients were about who would review the test

results and how these results would be used. (e) Aide- -how friendly, helpful and understanding the patients found the indi-

vidual who presented introductory instructions. (f) Involvement--how carefully and truthfully patients felt they behaved while re-

sponding to test items. (g) Need Assistance--how "impersonal" the test itself seemed to be and how the

patients perceived their need for help during the evaluation. (h) Relevance--how relevant test items seemed to be and how "silly" the patients

felt when taking the M M P I .

RESULTS AND DISCUSSION

Nonsignificant chi squares indicated that the computer and paper-and-pencil groups did not differ in terms of the distribution of men and women and the sources of referring clinical programs. Nonsignificant t-tests revealed that there were no dif- ferences between the two groups regarding age and scores on the 13 M M P I scales.

Measures of internal consistency (coefficient alpha; Cronbach, 1951) and corre-

114 Rozensky et al.

lations among the CAS scales are shown in Table 1. Five of eight alpha's were in the acceptable range for research tools (i.e., r >- .70), suggesting that these scales each represent reasonably coherent constructs. The three remaining alpha values were only marginal. Although a large percentage of the correlations between scales were statistically significant, most were quite low in magnitude, further suggest- ing that the scales represent more or less distinct constructs.

A 2 x 8 (groups by CAS scale) repeated measures analysis of variance yielded an overall significant difference between the two test-taking conditions (F [1,146] = 17.50; p < .0.001 ), with subjects in the computer group showing more overall positive attitudes toward taking the M M P I than their counterparts in the paper-and-pencil group. F tests comparing subjects' attitudes on each individual dimension further explain the overall positive attitude toward the computer- administered M M P [ (see Table 2). These analyses suggest that subjects in the corn-

Table 1. Computer Attitude Survey (CAS) Intercorrelations and Alpha Coefficients

1 2 3 4 5 6 7 8

1. Affect (.79) a 2. Cognitive .45" * (.75) 3. Time . 5 4 " * . 5 5 " * (.74) 4. Disposition . 2 0 * * - . 0 3 .02 (.79) 5. Aide . 2 6 * * , 5 9 * * . 3 9 * * - . 1 7 " * 6. Involvement .37* * .45* * .22* * - . 0 6 7. Need Assistance .31 * * .13" .32* * .08 8. Relevance . 1 9 " * .21 * * . 3 7 * * .05

(.70) .37" * (.55) .07 .02 (.55) .11 .09 . 2 7 " * (.55)

aNote: Coefficient alpha values are in parentheses. * p < . 0 5 * * p < . 0 1 .

Table 2. Dimensions of Attitude Responses I

Response Type (descriptors)

Type of Administration

Paper-and-Pencil Computer F (d.f.) 2

Affect* M 1.87 sd .80

Cognitive * * * M 2.09 sd .70

Time* * * M 2.39 sd .85

Disposition M 2.53 sd .92

Aide* * * M 1.98 sd .71

Involvement M 1.62 sd .66

Need Assis tance** M 2.26 sd .73

Relevance* * M 2.32 sd .81

1.65 3 .58 .61 (1,150}

1.71 12.62 .53 (1,165)

1.89 14.75 .74 (1,150)

2.46 .13 .93 (1,150)

1.60 11.72 .62 (1,150)

1.47 3.21 .51 (1,150)

2.02 6.12 .58 (1,165)

2.08 6 .12 .76 (1,165)

1Note: Lower scores reflect a more positive attitude. 2Note: Degrees of freedom vary due to missing cases. * p < . 0 6 . * * p < . 0 1 . * * * p < . 0 0 1 .

Patient attitudes towards the computerized MMPI 115

pute r group found the testing exper ience to be more interesting, more positive, less anxiety provoking and more relaxing than those in the paper-and-pencil group. Subjects who took the test on the compu te r also felt more positively toward the administrat ive aide and less in need of assistance dur ing the testing than those uti- lizing paper-and-penci l . Finally, those who took the test using the tradit ional approach felt that t ime went more slowly and the exper ience made "less sense" or was less relevant than those who were assigned to the compute r adminis t ra t ion group. T h e groups did not differ in terms of their concerns about their privacy or the disposition of test results, and both groups described themselves as equally involved in the testing experience.

T a k e n together , the results of this study suggest that patients ' at t i tudinal and affective responses of computer ized M M P I administrat ions may actually enhance ra ther than detract f rom the use of this valuable assessment tool. Patients in this s tudy seemed to feel more comfor table with au tomated testing than suggested by Lushene et al. (1974) and Hedl et al. (1973). Perhaps these differences reflect the fact that the prior studies used volunteer university students as subjects, had smaller sample sizes, and were conducted at a time when compute r acceptance in society was not as pervasive as today. In any case, the results of this investigation pro- vide encourag ing evidence that clinical populat ions do not rout inely suffer f rom cyberphob ia (P inker ton & Raffoul , 1984).

This invest igat ion purposely conta ined a heterogeneous sample of clinical sub- jects in order to investigate general attitudinal response of patients to the computer- ized administrat ion of the M M P I . Fur ther research is needed to determine whether there is an interact ion between patients ' diagnoses and their affective and attitu- dinal reactions to automated data collection. Future researchers should also replicate this s tudy with clinical ins t ruments other than the M M P I to de termine whether the positive findings of this s tudy can be generalized to other assessment tools.

REFERENCES

Adams, K.M., & Shore, D.L. (1976). The accuracy of an automated MMPI interpretation system in a psychiatric setting. Journal of Clinical Psychology, 32, 80-82.

Byers, A.P. (1981). Psychological evaluation by means of an on-line computer. Behavioral Research, Methods and Instrumentation, 12, 585-587.

Cogswell, J.F., & Estavan, D.P. (1965). Explorations in computer-assisted counseling. Santa Monica, CA: System Development Corporation.

Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

Dunn, T.G., Lushene, R.E., & O'Neil, H.F. (1972). Complete automation of the MMPI and a study of its response latencies. Journal of Consulting and Clinical Psychology, 39, 381-387.

Duthie, B. (1984). A critical examination of computer-administered psychological tests. In M.D. Schwartz (Ed.), Using Computers in Clinical Practice." Psychotherapy and Mental Health Applications, (pp. 135-139). New York: Haworth Press.

Elwood, D.L. (1969). Automation of psychological testing. American Psychologist, 24, 287-289. Elwood, D.L. (1972a). Automated WAIS testing correlated with face-to-face WAIS testing: A valid-

ity study. International Journal of Man-Machine Studies, 4, 129-137. Elwood, D.L. (1972b). Test-retest reliability and cost analyses of automated and face-to-face intel-

ligence testing. International Journal of Man-Machine Studies, 48, 1-22.

116 Rozensky et al.

Evan, W.M., & Miller, J.R. (1969). Differential effects on response bias of computer versus con- ventional administration of a social science questionnaire. An exploratory methodological exper- iment. Behavioral Science, 14, 216-227.

Finney, J.C. (1966). Programmed interpretation of MMPI and CPI. Archives of General Psychiatry, 15, 75-81.

Fowler, R.D. (1969). The current status of a computer interpretation of psychological tests. Ameri- can Journal of Psychiatry, 125, 21-27.

Hedl, J.J., O'Neil, H.F., & Hansen, D.N. (1973). Affective reactions toward computer-based intel- ligence testing. Journal of Consulting and Clinical Psychology, 40, 217-222.

Hofer, P.J. & Green, B.F. (1985). The challenge of competence and creativity in computerized psy- chological testing. Journal of Consulting and Clinical Psychology, 53, 826-838.

Katz, L. & Dalby, J.T. (1981). Computer and manual administration of the Eysenck Personality Inventory. Journal of Clinical Psychology, 37, 586-588.

Lachar, D. (1974a). Accuracy and generalizability of an automated MMPI interpretation system. Journal of Consulting and Clinical Psychology, 42, 267-273.

Lachar, D. (1974b). The MMPI." Clinical Assessment and Automated Interpretations. Los Angeles: West- ern Psychological Services.

Imshene, R.E., O'Neil, H.F., & Dunn, T. (1974). Equivalent validity of a completely computer- ized MMPI. Journal of Personality Assessment, 38, 353-361.

Orr, T.B. (1969). A comparison of the automated method and face-to-face method of administering the Wesch- ler Adult Intelligence Scale. Paper presented at Indiana Psychological Association.

Pinkerton, G.L., & Raftbul, P.R. (1984). Professional colleagues: Conflconting attitudes of proti:s- sionals towards micro-computers. In M.D. Schwartz (Ed.), Using Computers in Clinical Practice: P~~- chotherapy and Mental Health Application, (pp. 61-66). New York: Haworth Press.

Schwartz, M.D. (1984). Why do psychiatrists avoid using computers? In M.D. Schwartz (Ed.), UsbL# Computers in Clinical Practice." Psychotherapy and Mental Health Application, (p. 67). New" York: Haworth Press.

Slack, W.V., Hicks, G.P., Reed, C.E., & Van Cura, I..J. (1966). A computer-based medical-based history system. New England Journal of Medicine, 274, 194-198.

Slack, VV.V., & Van Cura, L.J. (1968). Patient reaction to computer based medical interviewing. Computers and Biomedical Research, 1, 527-531.

Smith, J.S. (1968). The computer and the TAT. Journal of School Psychology, 6, 206-214. Smith, R.E. (1963). Examination by computer. Behavioral Science, 8, 76-79.