Reliability of the histopathologic diagnosisof keratinocyte carcinomas

Jared Jagdeo, MS,a,b Martin A. Weinstock, MD, PhD,a,b,c Michael Piepkorn, MD,d

and Stephen F. Bingham, PhDe; The Department of Veteran Affairs

Topical Tretinoin Chemoprevention Trial Group

Providence, Rhode Island; Seattle, Washington; and Perry Point, Maryland

Objective: We sought to determine the interobserver reliability of the histopathologic diagnosis of basalcell carcinoma (BCC) and squamous cell carcinoma (SCC) (keratinocyte carcinomas) in the setting of aDepartment of Veteran Affairs multicenter chemoprevention study.

Methods: Interobserver concordance was assessed by blinded review of histopathologic slides by studydermatopathologists.

Results: Overall interobserver agreement between the two dermatopathogists was k = 0.69 (95%confidence interval [CI] 0.67-0.69). The dermatopathologists’ interobserver agreement was highest forbasal cell carcinoma at k = 0.88 (95% CI 0.84-0.91) and for a diagnostic category in the SCC-actinic kera-tosis spectrum at k = 0.80 (95% CI 0.73-0.86). The largest disagreements between the two referencedermatopathologists were regarding the categories of invasive SCC at k = 0.62 (95% CI 0.52-0.72), SCC insitu at k = 0.42 (95% CI 0.29-0.56), and actinic keratosis at k = 0.51 (95% CI 0.40-0.62). Agreement betweenthe local pathologists and central reference dermatopathologists were similar to the agreement between thecentral dermatopathologists. The morphea subtype of basal cell carcinoma was the only reliably diagnosedsubtype (k = 0.79, 95% CI 0.51-1.00), and tumor depth was reliably measured.

Limitations: A limitation of this study was the use of only two reference dermatopathologists.

Conclusion: Because of the impact on physician decision making and patient care, researchers andclinicians need to be aware of reliability of histopathology results, particularly pertaining to the SCC andactinic keratosis spectrum. ( J Am Acad Dermatol 2007;57:279-84.)

Keratinocyte carcinoma (KC) is the mostcommon malignancy in the United States,affecting more than one million people in

the United States per year. The two types of KC, basal

From the Dermatoepidemiology Unit, Providence Veterans Affairs

Medical Centera; Departments of Dermatologyb and Commu-

nity Health,c Brown Medical School, Providence; Division of

Dermatology, Department of Medicine, University of Washing-

tond; and The Veteran Affairs Cooperative Studies Program

Coordinating Center, Perry Point.e

Supported by Department of Veteran Affairs Office of Research

and Development, Cooperative Studies Program grant No. 402.

Conflicts of interest: None declared.

Accepted for publication March 28, 2007.

Reprints not available from the authors.

Correspondence to: Martin A. Weinstock, MD, PhD, Dermatoepi-

demiology Unit, Providence Veterans Affairs Medical Cen-

tere111D, 830 Chalkstone Ave, Providence, RI 02908. E-mail:

maw@brown.edu.

Published online May 10, 2007.

0190-9622/$32.00

ª 2007 by the American Academy of Dermatology, Inc.

doi:10.1016/j.jaad.2007.03.021

cell carcinoma (BCC) and squamous cell carcinoma(SCC) of the skin, have rapidly increased in incidenceduring the past several decades.1 Although therehave been multiple studies examining interobserverconcordance of melanocytic lesions, there have beenlimited reports of interrater reliability of KC. Wechose to explore this issue using data from a Depart-ment of Veteran Affairs Topical Tretinoin Chemopre-vention (VATTC) trial.

METHODSThe VATTC study was a randomized trial of topical

tretinoin 0.1% for the prevention of KC. The primary

Abbreviations used:

AK: actinic keratosisBCC: basal cell carcinomaCI: confidence intervalKC: keratinocyte carcinomaSCC: squamous cell carcinomaVATTC: Department of Veteran Affairs Topical

Tretinoin Chemoprevention

279

J AM ACAD DERMATOL

AUGUST 2007

280 Jagdeo et al

end point was the occurrence of a new BCC orinvasive SCC on the face or ears. All lesions biopsiedfrom the face or ears, regardless of clinical diagnosis,were processed for histopathologic analysis andstained with conventional hematoxylin-eosin. Eachslide was read by a local pathologist and one of twocentral reference dermatopathologists who wereblinded to the original diagnosis. The allocation ofeach slide to a central dermatopathologist wasdetermined by a computer-generated random num-ber. The diagnostic categories were: BCC, invasiveSCC, SCC in situ, actinic keratosis (AK), and other.The ‘‘other’’ category included a broad range ofdermatoses including seborrheic keratosis, seba-ceous hyperplasia, rosacea, nevomelonocytic nevus,fibrous papule, solar elastosis, and other pathology.The histologic subtype was also assessed for thosediagnosed as BCC and invasive SCC. Approximately10% of all specimens were randomly selected to beread by both central reference dermatopathologists.The central reference dermatopathologists were alsoasked to assess the quality of the slide.

The primary measure of interobserver agreement(interrater reliability) in this study was the k statistic,which measures the chance-corrected concordancebetween two raters.2,3 A k value of 1 indicates perfectagreement and a k statistic of 0 indicates only chanceagreement, with k values usually ranging between0 and 1. However, the k value can be negative. Anegative k value indicates that the agreement oc-curred less frequently than would be predicted bychance alone. Two scales and corresponding de-scriptive terminologies that have been proposed toevaluate k values and several are presented in Table I.

k Statistics were calculated for agreement on thediagnoses of the specimens that were evaluatedby both dermatopathologists, and for interobserveragreement between the local pathologist and the

Table I. Interpretations of k statistic values

Landis and Koch2

k Value range Descriptor\0.00 Poor0.00-0.20 Slight0.21-0.40 Fair0.41-0.60 Moderate0.61-0.80 Substantial0.81-1.00 Almost perfect

Fleiss3

k Value range Descriptor\0.40 Poor0.40-0.74 Fair to good0.75-0.99 Excellent1.00 Perfect

central reference dermatopathologist. We used k

statistics because they take into account agreementexpected by chance alone. If a slide had been readby both central reference dermatopathologists, thefirst evaluation was used for purposes of analysis ofagreement with the local pathologist. For purposesof this study, keratoacanthomas were consideredinvasive SCCs because of recent reports stating thatkeratoacanthomas are a variant of invasive SCC.4 Thek was calculated for the overall interoberver agree-ment using the 5 diagnostic categories. For analysisof interobserver agreement for each individual diag-nosis, the variables were then dichotomized, beforecalculation of the k statistics. The slide quality wasrated as acceptable, marginal, or poor.

Analyses were conducted with software (Stata,Version 8.2, StataCorp LP, College Station, Tex). Thisstudy was approved by the 8 relevant institutionalreview boards. All participants gave signed informedconsent at the outset of the study.

RESULTSA total of 3926 specimens were obtained from the

face and ears of participants during this trial. Allslides were read by local pathologists and one orboth central dermatopathologists. In all, 367 slideswere sent to both dermatopathologists, of which 355(97%) were assigned a diagnosis by both dermato-pathologists. There were 12 cases that were ex-cluded from analysis because the slide was lost orunavailable for one of the central dermatopatholo-gists (7 cases) or was judged uninterpretable as aresult of poor slide quality determined by one orboth of the readers (5 cases).

The largest disagreements between the two ref-erence dermatopathologists were regarding the cat-egories of SCC invasive (k = .62), SCC in situ (k = .42),and AK (k = .51) (Table II). However, agreement washigher when the squamous epithelial neoplasia(invasive and in situ SCC and AK) were groupedtogether as a single diagnosis (k = .80, 95% confi-dence interval [CI] .73-.86).

Dermatopathologist reader 1 read a higher per-centage of the same slides as SCC invasive comparedwith dermatopathologist reader 2 (26% vs 14%, P \.0001) (Table III).

Better interobserver agreement was demonstratedin BCC (k = .88). These k statistics were strikinglysimilar when comparing agreement between thecentral reference dermatopathologists and the agree-ment between the local pathologists and the centraldermatopathologists. The disagreements betweenthe reference dermatopathologists and local pathol-ogists were similar to those observed between thetwo reference dermatopathologists (Table III).

J AM ACAD DERMATOL

VOLUME 57, NUMBER 2

Jagdeo et al 281

Table II. Interobserver agreement

k 95% CI Agreement, %

On diagnosis between reference dermatopathologistsAll diagnoses 0.69 0.67-0.69 76BCC 0.88 0.84-0.91 94SCC invasive 0.62 0.52-0.72 88SCC in situ 0.42 0.29-0.56 93Actinic keratosis 0.51 0.40-0.62 85Other 0.71 0.62-0.81 91

On diagnosis between local pathologistsand central reference dermatopathologists

All diagnoses 0.66 0.66-0.67 75BCC 0.90 0.88-0.91 95SCC invasive 0.60 0.57-0.64 90SCC in situ 0.40 0.34-0.45 93Actinic keratosis 0.51 0.48-0.55 84Other 0.61 0.58-0.64 88

On BCC histologic subtypeAll histologic types 0.36 0.30-0.38 63Morphea 0.79 0.51-1.00 98Infiltrating/micronodular 0.17 �0.03-0.37 77Superficial multicentric 0.41 0.18-0.63 86Basal with squamous differentiation * * 97Fibroepithelioma * * 93Nodular 0.38 0.24-0.51 69

On invasive SCC histologic subtypeAll histologic types 0.34 0.24-0.45 63Spindle cell * * 98Verrucous * * 83Acantholytic 0.46 0.20-0.72 74Other type 0.37 0.12-0.63 68

On invasive SCC gradeAll grades 0.25 0.20-0.32 62Well differentiated 0.23 �0.02-0.47 60Moderately differentiated 0.20 �0.06-0.47 60Poorly differentiated * * 91

BCC, Basal cell carcinoma; CI, confidence interval; SCC, squamous cell carcinoma.

*At least one rater designated less than 3 specimens in this category.

High interrater agreement was noted for themorphea subtype of BCC. The other BCC histologictypes, and the subtypes and grade of the SCCsdemonstrated low interobserver k values, althoughthe precision of these estimates were generally low.There was good agreement, however, on whetherdepth was assessable for BCC (k = 0.85, 95% CI .76-.95) and invasive SCC (k = 0.73, 95% CI .50-.95).Among assessable specimens, Pearson correlationswere high between dermatopathologists for mea-sured depth in millimeters (0.97 for BCC [n = 41] and0.95 for SCC [n = 10]).

The concordance on diagnosis between the localpathologist and the central dermatopathologist wassimilar among each of the 6 clinical centers of thistrial. Furthermore, the concordance between refer-ence dermatopathologist 1 and the local pathologists

was similar to the concordance between referencedermatopathologist 2 and local pathologists.

DISCUSSIONIn this study we found a high degree of interrater

agreement of the diagnosis of BCC. The interrateragreement of the diagnosis of SCC was not as high.There were disagreements among invasive SCC, SCCin situ, and AK. We also found the morphea subtypeof BCC was reliably diagnosed, as was measurementof tumor depth for both BCC and SCC. The otherhistologic subtypes of BCC and SCC and the gradeof SCC showed fair to poor interrater agreement.Agreement for diagnosis between two referencedermatopathologists was similar to the interrateragreement between reference dermatopathologistsand pathologists who initially read the slides.

J AM ACAD DERMATOL

AUGUST 2007

282 Jagdeo et al

Table III. Histopathologic classification of lesions by different readers

Diagnosis comparison between reference dermatopathologists

Dermatopathologist 2

Diagnosis BCC Invasive SCC SCC in situ AK Other Total

Dermatopathologist 1 BCC 116 0 0 0 3 119 (34%)Invasive SCC 9 51 6 19 6 91 (26%)SCC in situ 1 0 10 16 1 28 (8%)AK 2 0 0 40 11 53 (15%)Other 5 0 0 6 53 64 (18%)Total 133 (37%) 51 (14%) 16 (5%) 81 (23%) 74 (21%) 355 (100%)

Diagnosis comparison between reference dermatopathologists and local pathologists

Local pathologists

Diagnosis BCC Invasive SCC SCC in situ AK Other Total

Reference dermatopathologists BCC 1469 16 7 4 22 1518 (39%)Invasive SCC 46 363 33 96 41 579 (15%)SCC in situ 7 50 108 94 19 278 (7%)AK 37 70 54 511 135 807 (20%)Other 55 24 17 149 499 744 (19%)Total 1614 (41%) 523 (13%) 219 (6%) 854 (22%) 716 (18%) 3926 (100%)

Comparison between reference dermatopathologists assessment of slide quality

Reference dermatopathologist 2

Quality of Slide Acceptable Marginal Poor Total

Reference dermatopathologist 1 Acceptable 224 120 1 345 (96%)Marginal 2 9 4 15 (4%)Poor 0 0 0 0 (0%)Total 226 (63%) 129 (36%) 5 (1%) 360 (100%)

Comparison between reference dermatopathologists assessment of BCC subtype

Dermatopathologist 2

BCC histologic

type Morphea

Infiltrating/

micronodular

Superficial

multicentric

Basal with

squamous

differentiation

Fibro-

epithelioma Nodular Total

Dermatopathologist 1 Morphea 4 0 0 0 0 0 4 (3%)Infiltrating/

micronodular2 5 0 4 0 0 11 (10%)

Superficialmulticentric

0 0 7 0 0 1 8 (7%)

Basal withsquamousdifferentiation

0 0 0 1 0 0 1 (1%)

Fibroepithelioma 0 0 0 0 0 0 0 (0%)Nodular 0 20 15 0 0 56 91 (79%)Total 6 (5%) 25 (22%) 22 (19%) 5 (4%) 0 (0%) 57 (50%) 115 (100%)

Comparison between reference dermatopathologists assessment of SCC subtype

Dermatopathologist 2

Invasive SCC

histologic type Spindle cell Verrucous Acantholytic Other type Total

Dermatopathologist 1 Spindle cell 0 0 1 0 1 (2%)Verrucous 0 0 1 0 1 (2%)Acantholytic 0 1 12 4 17 (36%)Other type 0 6 5 17 28 (60%)Total 0 (0%) 7 (15%) 19 (40%) 21 (45%) 47 (100%)

J AM ACAD DERMATOL

VOLUME 57, NUMBER 2

Jagdeo et al 283

Table III. Cont’d

Comparison between reference dermatopathologists assessment of differentiation of SCC

Dermatopathologist 2

Invasive SCC

grade

Well

differentiated

Moderately

differentiated

Poorly

differentiated Total

Dermatopathologist 1 Well differentiated 12 13 2 27 (58%)Moderately differentiated 4 12 2 18 (38%)Poorly differentiated 0 0 2 2 (4%)Total 16 (34%) 25 (53%) 6 (13%) 47 (100%)

AK, Actinic keratosis; BCC, basal cell carcinoma; SCC, squamous cell carcinoma.

Limited studies exist regarding the interobserveragreement for diagnosis ofKC. This contrastswith thenumerous published reports examining the interob-server concordance of melanocytic lesion studies.5-12

A reproducibility study of histopathologic diag-nosis of KC was conducted in the setting of theHELIOS study (a multicenter southern Europeancase-control study).13 From 1989 to 1992, theHELIOS group obtained slides of 1774 routine diag-noses of KC in individuals 20 to 70 years of ageidentified from designated southern European can-cer registries and hospitals. In the HELIOS study,each specimen was reviewed by a reference pathol-ogist at the local center and then blindly rereviewedby one of 10 reference pathologists. The lesionsanalyzed were located anywhere on the body, andthey evaluated only lesions previously identified asKC, whereas our study evaluated all biopsied lesionson the face and ears, regardless of benign or malig-nant diagnosis. In the context of HELIOS, less than2% of specimens were reclassified as benign, and thek for that determination between reference pathol-ogists was 0.85 (95% CI 0.77-0.94). Among thoseconfirmed as malignant, the k for both BCC and SCCwas 0.88 (CI 0.84-0.91 and 0.85-0.92, respectively).Among those confirmed as SCC, the k for thediagnosis of invasive SCC was 0.67 (CI 0.56-0.79).Among those confirmed as BCC, the k for themorpheaform subtype was 0.56 (CI 0.50-0.62). Theselection of lesions for histopathologic review maybe a key difference with the VATTC trial, althoughdiagnostic criteria may also have differed.

Another study evaluated 15 slides chosen torepresent a diagnostic spectrum from AK to SCC.These slides were reviewed by 77 of the first 100registrants at a dermatopathology conference, whowere asked to classify each as an AK or SCC. Theintraclass correlation coefficient for that determina-tion was 0.97.14

Our study was limited by the use of only tworeference dermatopathologists who may or may notbe representative. These two dermatopathologists

did not have periodic reinforcement of diagnosticcriteria during the 6-year course of the VATTC trial,which, if performed, might have resulted in betteragreement. We did not assess the reading of theindividual lesion histologic characteristics that leadto a diagnosis, which would have allowed us tobetter define potential avenues to increase reliability.In addition, our study was limited to lesions of theface and ears in individuals at high risk, so general-izability to other contexts is unproven. We examinedall lesions biopsied from those locations in a blindedmanner. Furthermore, we not only examined theagreement between our reference dermatopatholo-gists, but also the concordance of our local pathol-ogists who were not part of the study team with ourreference dermatopathologists.

The spectrum of invasive SCC, SCC in situ, and AKis well known but the boundaries of these conditionsare imprecisely defined, and the reliability of thedistinctions is limited. Our results reflect this diag-nostic difficulty. Indeed, there are no sharp histo-logic criteria that precisely define the limits betweenAK and early SCC in situ, or between a hypertrophicAK and microinvasive carcinoma and the discrimi-nations are subjective. Furthermore, there is litera-ture asserting that ‘‘actinic keratosis is squamous cellcarcinoma,’’ a conceptual paradigm that could intro-duce difficulty in assignment of diagnosis by derma-topathologists who subscribe to that viewpoint.15,16

In addition, the reliability of the SCC histologicsubtype and grading have important clinical impli-cations, because the poorly differentiated SCCs aremost likely to recur and metastasize.17-19

In practice there is potentially broad overlapamong nodular, micronodular, and infiltrative BCCsand drawing boundaries among them may be arbi-trary. The reliability of subtype classifications ofBCCs is important because of the varied potentialprognostic outcomes. BCCs rarely metastasize, butsome have invasive qualities. In particular, the infil-trative, micronodular, superficial, and morpheaformBCCs are more aggressive subtypes and recur at

J AM ACAD DERMATOL

AUGUST 2007

284 Jagdeo et al

higher rates than other subtypes.20,21 The increase inrecurrence rates of these subtypes have been attrib-uted to their incomplete excision without the tumorbeing visible in the margins evaluated.21 The varia-bility in histologic diagnosis shown in this analysisunderscores the fact that histologic diagnoses arehypotheses about the nature of diseases in patientsas inferred from observations. As such, histopatho-logic opinions can, and often do, legitimately differ.

The decision to treat a lesion with a topicalmedication, cryotherapy, or excisional or Mohs mi-crographic surgery sometimes rests on these histo-logic opinions that are less than perfectly reliable.As dermatologists, we should factor this into ourdecision-making process, and arrive at our recom-mendations with appropriate humility. Given thepotential clinical implications of these diagnoses, ourresults emphasize the need to improve the reliabilityof these diagnoses, and suggest that researchers andclinicians need to be cognizant of assessing reliabilityof histopathology results.

The authors thank Kimberly Marcolivio, project coordi-nator, for her administrative assistance, Steven Reinert forhis software assistance, and Clifton White, MD, for hisdermatopathology assistance.

REFERENCES

1. Karagas MR, Weinstock MA, Nelson HH. Keratinocyte carcino-

mas (basal cell and squamous cell carcinomas of the skin).

In: Schottenfeld D, Fraumeni JF, editors. Cancer epidemiology

and prevention. 3rd ed. New York: Oxford University Press;

2006. pp. 1230-50.

2. Landis JR, Koch GG. The measurement of observer agreement

for categorical data. Biometrics 1977;33:159-74.

3. Fleiss JL. Statistical methods for rates and proportions. New

York: Wiley; 1981. p. 218.

4. Kane CL, Keehn CA, Smithberger E, Glass LF. Histopathology

of cutaneous squamous cell carcinoma and its variants. Semin

Cutan Med Surg 2004;23:54-61.

5. Clemente C, Cochran AJ, Elder DE, Levene A, MacKie RM, Mihm

MC, et al. Histopathologic diagnosis of dysplastic nevi: con-

cordance among pathologists convened by the World Health

Organization melanoma program. Hum Pathol 1991;22:313-9.

6. de Wit PE, van’t Hof-Grootenboer B, Ruiter DJ, Bondi R, Brocker

EB, Cesarini JP, et al. Validity of the histopathological criteria

used for diagnosing dysplastic nevi: an interobserver study by

the pathology subgroup of the EORTC malignant melanoma

cooperative group. Eur J Cancer 1993;29A:831-9.

7. Duncan LM, Berwick M, Bruijn JA, Byers HR, Mihm MC, Barnhill

RL. Histopathologic recognition and grading of dysplastic

melanocytic nevi: an interobserver agreement study. J Invest

Dermatol 1993;100(Suppl):318S-21S.

8. Meyer LJ, Piepkorn M, Goldgar DE, Lewis CM, Cannon-Albright

LA, Zone JJ, et al. Interobserver concordance in discriminating

clinical atypia of melanocytic nevi, and correlations with

histologic atypia. J Am Acad Dermatol 1996;34:618-25.

9. Pozo L, Naase M, Cerio R, Blanes A, Diaz-Cano SJ. Critical

analysis of histologic criteria for grading atypical (dysplastic)

melanocytic nevi. Am J Clin Pathol 2001;115:194-204.

10. Annessi G, Cattaruzza MS, Abeni D, Baliva G, Laurenza M, Macchini

V, et al. Correlation between clinical atypia and histologic

dysplasia in acquired melanocytic nevi. J Am Acad Dermatol

2001;45:77-85.

11. Weinstock MA, Barnhill RL, Rhodes AR, Brodsky GL; Dysplastic

Nevus Panel. Reliability of the diagnosis of melanocytic

dysplasia. J Cutan Pathol 1996;23:64.

12. Weinstock MA, Barnhill RL, Rhodes AR, Brodsky GL; Dysplastic

Nevus Panel. Reliability of the diagnosis of melanocytic

dysplasia. Arch Dermatol 1997;133:953-8.

13. Wechsler J, Zanetti R, Schrameck C, Rosso S, Pippione M,

Linares J, et al; HELIOS Group. Reproducibility of histo-

pathologic diagnosis and classification of non-melanocytic

skin cancer: a panel exercise in the framework of the multi-

center southern European study HELIOS. Tumori 2001;87:95-

100.

14. Davis DA, Donahue JP, Bost JE, Horn TD. The diagnostic

concordance of actinic keratosis and squamous cell carci-

noma. J Cutan Pathol 2005;32:546-51.

15. Ackerman AB, Mones JM. Solar (actinic) keratosis is squamous

cell carcinoma. Br J Dermatol 2006;155:9-22.

16. Cockerell CJ. Histopathology of incipient intraepidermal squa-

mous cell carcinoma (‘‘actinic keratosis’’). J Am Acad Dermatol

2000;42:11-7.

17. Lohmann CM, Solomon AR. Clinicopathologic variants of

cutaneous squamous cell carcinoma. Adv Anat Pathol 2001;

8:27-36.

18. Petter G, Haustein UF. Histologic subtyping and malignancy

assessment of cutaneous squamous cell carcinoma. Dermatol

Surg 2000;26:521-30.

19. Cassarino DS, Derienzo DP, Barr RJ. Cutaneous squamous cell

carcinoma: a comprehensive clinicopathologic classificatione

part two. J Cutan Pathol 2006;33:261-79.

20. Scrivener Y, Grosshans E, Cribier B. Variations of basal cell

carcinomas according to gender, age, location and histopath-

ological subtype. Br J Dermatol 2002;147:41-7.

21. Raasch BA, Buettner PG, Garbe C. Basal cell carcinoma: histo-

logical classification and body-site distribution. Br J Dermatol

2006;155:401-7.