Contribution of the Diagnostic Test to thePhysician’s Diagnostic Thinking:

New Method to Evaluate the Effect1

Yoshito Tsushima, MD, Jun Aoki, MD, Keigo Endo, MD

Rationale and Objectives. The purpose of this study was to develop and test a method for evaluating the effect of diag-nostic tests on physicians’ diagnostic thinking.

Materials and Methods. The authors applied the method to data obtained in a previous study. The diagnoses of surgeonsin the emergency department before and after contrast material–enhanced computed tomography (CT) and their certaintywere collected from the cases of 125 adult patients with acute abdominal pain. When the pre- or post-CT diagnosis wasnot consistent with the final diagnosis, the diagnostic certainty reported by a physician was converted to a negative value(true diagnostic certainty). Change in true diagnostic certainty was calculated by subtracting the values for diagnostic cer-tainty before CT from those after CT.

Results. In 118 cases, the change in true diagnostic certainty was positive, which indicates that the CT information wasbeneficial for the physician’s diagnostic thinking. The true diagnostic certainty before CT was 32.9%� 57.6 (range:�90% to 100%; 95% confidence interval: 22.7%, 43.1%) and that after CT was 73.4%� 37.0 (range:�80% to 100%;95% confidence interval: 66.8%, 80.0%;P � .0001). Thus, the increase in true diagnostic certainty was 40.5%� 58.8(range:�160% to 170%; 95% confidence interval: 30.1%, 69.2%).

Conclusion. This method may be useful for estimating the effect of diagnostic tests on physicians’ diagnostic thinking invariable conditions.

Key Word. Computed tomography (CT), clinical effectiveness.

© AUR, 2003

We recently measured the effect of computed tomo-graphic (CT) findings on physicians’ diagnosis and treat-ment plan in patients with acute abdominal pain (1); weused the method proposed by Thornbury et al (2) andmodified by Omary et al (3). In that study, the physicianswere asked to identify one most likely diagnosis and its

probability (diagnostic certainty) before information fromcontrast material–enhanced CT was given to them. TheCT information was then presented and the question re-peated. In 125 adult patients presenting with acute ab-dominal pain (1), CT information changed the initial di-agnosis in 40 (32%). The diagnostic certainty was 58.3%before CT information was received, and it increased by21.9% afterward (P � .0001). These results suggestedthat contrast-enhanced CT was an important tool for theemergency department physicians.

However, there were several limitations in the Thorn-bury-Omary method to evaluate the effect of the diagnos-tic test on physicians’ diagnostic thinking. First, the cal-culation of change in diagnostic certainty does not take

Acad Radiol 2003; 10:751–755

1 From the Department of Radiology, Motojima General Hospital, 3-8 Nishi-Honcho, Ohta, Gunma 373-0033, Japan (Y.T.), and the Department of Di-agnostic Radiology and Nuclear Medicine, Gunma University Hospital, Ja-pan (J.A., K.E.). Received March 5, 2003; revision requested March 20; fi-nal revision received April 4; accepted April 7. Address correspondenceto Y.T.

© AUR, 2003

751

into account whether the pre- or posttest diagnosis is con-sistent with the final diagnosis (ie, whether the pre- orposttest diagnosis is correct). Second, an increase in diag-nostic certainty may not equate with an increased benefitto the patient, since a change in the pretest diagnosis orin treatment plans could be counterproductive. Finally, theresults may underestimate the true effect of the diagnostictest on diagnoses, since the inferred pretest diagnosticcertainty tends to affect the true certainty percentages.

Figure 1shows the calculation of diagnostic certaintyin three cases in which CT was performed in patientswith acute abdominal pain. In case 1, the inferred diag-nostic certainty before CT for the post-CT diagnosis isestimated to be the same as the certainty (40%) given bythe physician because the diagnostic certainty before CTis less than 50%. Thus, the increase in diagnostic cer-tainty is positive (70%� 40% � 30%). Since thepost-CT diagnosis is consistent with the final diagnosis(ie, acute appendicitis) and the certainty of the post-CTdiagnosis is greater than that of the pre-CT diagnosis, theCT information is beneficial for the physician’s diagnosticthinking. In case 2, however, the post-CT diagnosis iscolon diverticulitis, and the CT information is harmful(counterproductive). Since with this method the increasein diagnostic certainty does not depend on the final diag-nosis, the CT information is considered to be beneficialfor the physician’s diagnostic thinking, even when thepost-CT diagnosis is incorrect.

In case 3, the diagnostic certainty increases by 30%(70% � 40% � 30%), as in case 1. However, we believethe CT information has a greater effect on the physician’sdiagnostic thinking in case 1 than it does in case 3, sincethe diagnosis itself is changed in case 1, possibly result-ing in a change of treatment.

To conquer the limitations of the Thornbury-Omarymethod, we proposed a new method and applied it inthe present study to the data obtained in our previousstudy (1).

MATERIALS AND METHODS

Evaluation Method

With the Thornbury-Omary method, the effect of thediagnostic test is assessed by evaluating the percentagechange in diagnostic certainty without consideringwhether the pre- and posttest diagnoses are consistentwith the final diagnosis. We modified this method.

When the pre- or posttest diagnosis was not consistentwith the final diagnosis, the diagnostic certainty reportedby a physician was converted to a negative value. Wedefined this adjusted value as “true diagnostic certainty.”For instance, if the diagnosis was not consistent with thefinal diagnosis and its diagnostic certainty was 20%, thetrue diagnostic certainty was�20%.

Next, the change in true diagnostic certainty was cal-culated by subtracting the values of the certainty beforethe test from those after the test (Fig 2). Six patterns areassumed, according to whether the pre- or posttest diag-nosis was consistent with the final diagnosis and accord-ing to the direction and magnitude of the change in truediagnostic certainty after the test.

Patients and Data AcquisitionIn this study, we applied our new method to an analy-

sis of data obtained in a previous study (1). In the previ-ous study, one of four experienced surgeons evaluated alladult patients (�18 years old) who were admitted to theemergency department of our 350-bed community hospi-tal with acute abdominal pain in the course of 1 year. ACT examination was indicated when any condition thatcould require surgical intervention was clinically sus-

Figure 1. Calculation of change in diagnostic certainty with theThornbury-Omary method. In all of these cases, the increase indiagnostic certainty resulting from CT information is of the samemagnitude (30%) because the change in diagnostic certainty isindependent of the final diagnosis (ie, it is the same regardless ofwhether the pre- and post-CT diagnoses are correct).

TSUSHIMA ET AL Academic Radiology, Vol 10, No 7, July 2003

752

pected or when the surgeon expressed difficulty in defin-ing the cause. A total of 125 CT examinations (125 adultpatients [74 men and 51 women; mean age, 40 years�19.3; age range, 18–92 years]) were performed, and theresultant data are included in this study.

Contrast-enhanced CT of the whole abdomen was per-formed in all patients. A radiologist with knowledge ofthe patient’s signs and symptoms and of the results oflaboratory tests performed as part of the daily clinicalroutine immediately interpreted these CT images. Theemergency department physicians were asked to identifyone most likely diagnosis (pre-CT diagnosis) and to esti-mate the certainty (0%–100%) that the initial diagnosiswas correct. Immediately after reviewing the CT imagesand the radiologic report, the same surgeon was asked toprovide a revised diagnosis (post-CT diagnosis) and toestimate the certainty of this diagnosis. The final diag-noses were determined at surgery and pathologic exami-nation for the inpatients who underwent laparotomy (n �54). If the patients were admitted to an inpatient ward butdid not undergo surgery, the most likely discharge diag-nosis was defined as the final diagnosis (n � 61). Allpatients who were discharged home with or without medi-

cation were followed up and the final diagnoses were de-termined in the outpatient clinic (n � 10).

Statistical AnalysisResults were expressed as the mean� standard devia-

tion (SD). The Studentt test was used, andP values lessthan .05 were considered to indicate a statistically signifi-cant difference.

RESULTS

The pre- and post-CT diagnoses were compared withthe final diagnosis, and the results are summarized in theTable. In 118 cases, the increase in true diagnostic cer-tainty was positive; thus the CT information was consid-ered beneficial for the physician’s diagnostic thinking(productive results).

The true diagnostic certainty before CT was 32.9%�57.6 (range:�90% to 100%; 95% confidence interval:22.7%, 43.1%) and that after CT was 73.4%� 37.0(range:�80% to 100%; 95% confidence interval: 66.8%,80.0%;P � .0001). Thus, the increase in true diagnostic

Figure 2. The authors’ construct includes six possible patterns of change in diagnostic certainty: (a–c) three in which the change isproductive and (d–f) three in which it is counterproductive. Each pattern varies according to whether the pre- and posttest diagnoses areconsistent with the final diagnosis and as a function of the change in true diagnostic certainty after the diagnostic test. Change in truediagnostic certainty is calculated by subtracting the value of true diagnostic certainty before the test from that after the test. The horizon-tal bars indicate the magnitude of true diagnostic certainty. (a) Both pre- and posttest diagnoses are correct, and there is an increase intrue diagnostic certainty. (b) The pretest diagnosis is incorrect, but the posttest diagnosis is correct; the result, again, is an increase intrue diagnostic certainty. (c) Both pre- and posttest diagnoses are incorrect, but true diagnostic certainty nevertheless is increased.(d) Both pre- and posttest diagnoses are correct, but true diagnostic certainty is decreased—a counterproductive result. (e) The pretestdiagnosis is correct, but the posttest diagnosis is incorrect; again, the test is counterproductive. (f) Both pre- and posttest diagnosesare incorrect, and true diagnostic certainty is decreased—the most counterproductive result.

Academic Radiology, Vol 10, No 7, July 2003 CONTRIBUTION OF THE DIAGNOSTIC TEST

753

certainty was 40.5%� 58.8 (range:�160% to 170%;95% confidence interval: 30.1%, 69.2%).

DISCUSSION

Fryback and Thornbury (4) described a six-tiered hier-archic model for the assessment of diagnostic imagingtests: technical efficacy (level 1), diagnostic accuracy(level 2), effect on diagnostic thinking (level 3), therapeu-tic efficacy (level 4), effect on patient outcome (level 5),and societal efficacy (level 6). There have been manystudies on levels 1 and 2; for instance, accuracy of diag-nosis, predictive values for positive and negative exami-nations, sensitivity, specificity, and measures relating toreceiver operating characteristic (ROC) curves have beencalculated. Studies on the efficacy of diagnostic thinking(level 3) determine the effect of imaging information onthe diagnostic thinking of the physician. This informationmay reassure the physician, alter the relative order of thedifferential diagnosis, or indicate an entirely new diagno-sis. However, it is not easy to estimate the effect. Thorn-bury et al (2) and Fryback et al (4) suggested the log-likelihood ratio as one method of analyzing data in stud-ies of this type. Omary et al (3) simplified their methodand used the difference in physician’s subjective probabil-ity estimates for a given diagnosis. Although this methodis useful, there are inevitable drawbacks, as discussed inthe introduction to the present article.

We proposed a new method for estimating the effectof the diagnostic test, and we applied it to contrast-en-hanced CT in patients with acute abdominal pain. Withthis method, the diagnoses before and after imaging arecompared with the final diagnosis (the standard). An in-correct diagnosis is considered to indicate a negative ef-fect of the test on the physician’s diagnostic thinking(counterproductive effect). A physician is asked to iden-tify only one most likely diagnosis and to estimate itscertainty before and after the test. Therefore, in theThornbury-Omary method, in order to calculate the gainin diagnostic certainty resulting from the diagnostic test,it is necessary to infer the certainty before the test for theposttest diagnosis when the physician’s diagnosis beforethe test differs from that after the test. In our newmethod, this inference is not necessary.

With our new method, a positive increase in true diag-nostic certainty shows that the diagnostic test generallyprovides a beneficial effect on the physician’s diagnosticthinking, enabling the physician to reach the correct diag-nosis. In case 1 (Fig 1), the gain in true diagnostic cer-

tainty is calculated as 110% (70%� [�40%] � 110%),representing a beneficial effect of contrast-enhanced CT—much higher than the gain in true diagnostic certainty incase 3 (70%� 40% � 30%). In case 2, the increase intrue diagnostic certainty is�110% (�70% � 40% ��110%), representing a harmful effect on the physician’sdiagnostic thinking.

In this study, the average true diagnostic certainty ofthe post-CT diagnosis was 73.4%—significantly in-creased, compared with that of the pre-CT diagnosis.Thus, we can conclude that contrast-enhanced CT wasuseful for diagnosis in most patients with acute abdominalpain, despite several cases in which it was counterproduc-tive.

Yet, even this new method has its limitations. Diag-nostic certainty as estimated by physicians is a continuousor integral category of measure. Therefore, a change of20% in the certainty of diagnosis for a single patient isexactly the same in value as changes of 10% in diagnos-tic certainty for two patients. Furthermore, on a true diag-nostic certainty scale, a positive change of 10% is equalin magnitude to a negative change of 10% and a changefrom 10% to 20% is of exactly the same magnitude as achange from 80% to 90%. These assumptions of equiva-lence, although mathematically accurate, are not necessar-ily valid in the clinical setting. Second, measurementscales such as this one can have important biases relatedto scaling and anchoring effects. For instance, individualstend to avoid the extremes of a scale (ie, 0% and 100%)when asked to assess subjective factors. Also, the posttestdiagnosis may be influenced by the pretest diagnosis.

Despite the necessity of further exploration of theseissues, we believe that this method may be useful for esti-

Results of Calculation of True Diagnostic Certainty

Overall Effect ofCT on Diagnosis*

Diagnosisbefore CT

Diagnosisafter CT

No. of Cases(n � 125)

Productivea Correct Correct 84b Incorrect Correct 32c Incorrect Incorrect 2

Counterproductived Correct Correct 0e Correct Incorrect 5f Incorrect Incorrect 2

Note.—For diagnoses before CT and after CT, correct means“consistent with the final diagnosis” and incorrect means “incon-sistent with the final diagnosis.”* Letters refer to parts of Figure 2.

TSUSHIMA ET AL Academic Radiology, Vol 10, No 7, July 2003

754

mating the effect of the diagnostic test on physicians’diagnostic thinking in variable conditions.

REFERENCES

1. Tsushima Y, Yamada S, Aoki J, Motojima T, Endo K. Effect of contrast-enhanced computed tomography on diagnosis and management ofacute abdomen in adult. Clin Radiol 2002; 57:507–513.

2. Thornbury JR, Fryback DG, Edwards W. Likelihood ratios as a measureof the diagnostic usefulness of excretory urogram information. Radiol-ogy 1975; 114:561–565.

3. Omary RA, Kaplan PA, Dussault RG, et al. The impact of ankle radio-graphs on the diagnosis and management of acute ankle injuries. AcadRadiol 1996; 3:758–765.

4. Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. MedDecis Making 1991; 11:88–94.

Academic Radiology, Vol 10, No 7, July 2003 CONTRIBUTION OF THE DIAGNOSTIC TEST

755