Updates to and Cross-Reader Reliability Validation of the VASARI Brain Tumor Feature Set for Distributed Scoring of Low-Grade Brain Tumor Cases in Radiogenomic

Updates to and Cross-Reader Reliability Validation of the VASARI Brain Tumor Feature Set for Distributed Scoring of Low-Grade Brain Tumor Cases in Radiogenomic Research

The TCGA Low-Grade Glioma Group, including, but not limited to

Chen JY, Gutman DA, Yeom KW, Mukherjee S, Iv M, Wintermark M, Wolansky L, Griffith B, Holder CA, Hwang SN, Colen R, Kirby J, Jaffe CC, Fevrier-Sullivan B, and Flanders AE

eEdE-205Control: 1771

Disclosures

• None

VASARI Features Introduction

VASARI terminology– Categorical descriptors of MR imaging features of

human gliomas– Based on previous REMBRANDT project• REpository for Molecular BRain Neoplasia DaTa

– Bioinformatics framework leveraging data warehousing for clinical and functional genomic data from glioma clinical trials

Sept 2013 VASARI Feature SetFeature number/name

Description Options

F0 Image QA Quality assurance pre-check. Inventory of required image series including FLAIR/T2 and pre/post contrast T1WI.

Post biopsy (requires adjudication)

Post-op study (disqualified) No contrast injected

(disqualified) No T2/FLAIR images

(disqualified)

F1 Tumor Center Location of lesion geographic center; the largest component of the tumor (either CET or nCET). (multiple selections acceptable – choose up to two)

Frontal lobe Temporal lobe Parietal lobe Occipital lobe Insula Brainstem Cerebellum Lentiform nucleus Caudate nucleus Thalamus Corpus callosum

F34 Tumor Origin Propose where you think the tumor originated from using the following grades. Periventricular (most likely arising from periventricular/subventricular zone); Gyral (most likely arising from the cortex or adjacent gyral subcortical white matter zone);Deep white matter (most likely arising deep to the U fiber region but not in the periventricular zone). Use not applicable if none of these feature criteria apply. (multiple selections acceptable)

Periventricular Gyral Deep white matter Other

Sept 2013 VASARI Feature SetF2 Side of TumorCenter

Side of lesion center irrespective of whether the lesion crosses into the contralateral hemisphere.

Right Center/Bilateral Left

F3 Eloquent Brain Does any component of the tumor (CET or nCET) involve eloquent cortex or the immediately adjacent subcortical white matter of eloquent cortex (motor,language, vision)? (multiple selections acceptable)

No Eloquent Brain Speech motor Speech receptive Motor Vision

F22 nCET tumorCrosses Midline:

nCET tumor crosses midline is defined by any nCET tumor that extends into the contralateral hemisphere through white matter commissures usually expected at the midline (exclusive of herniated ipsilateral tissue).

No nCET No Yes

F13 Definition of thenon-enhancingmargin (e.g.Grade III)

If most of the outside non-enhancing (nCET) margin of the tumor is well-defined (i.e. sharply marginated) and smooth (geographic), versus if the margin is poorly-defined (fluffy or indistinct).

Completely well-defined (100%)

Mostly well-defined (> two thirds)

Mixed (@ 50-50) Mostly poorly-defined (> two

thirds) Completely ill-defined (100%) No nCET

Sept 2013 VASARI Feature SetF6 Proportion nCET When scanning through the entire tumor volume,

what proportion of the entire tumor is estimated to represent nonenhancing tumor or nCET (not edema)? Non-enhancing tumor (nCET) is defined as regions of T2W intermediate hyperintensity (less than the intensity of cerebrospinal fluid or vasogenic edema, with corresponding T1W hypointensity) that are associated with mass effect and architectural distortion, including blurring of the gray-white interface. (This may be difficult to discern from vasogenic edema – if uncertain error conservatively). (Assuming that the the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) a edema component.)

> Two Thirds Between Two Thirds and

One Third Less than One Third Minimal None

Sept 2013 VASARI Feature Set

F14 Proportion ofEdema

Visually, when scanning through the entire tumor volume, what proportion of the entire abnormality is estimated to represent vasogenic edema? Edema should be greater in signal than nCET and somewhat lower in signal than CSF on T2W or T2W FLAIR. Pseudopods are characteristic of edema extending up to the subcortical white matter but not infiltrating gray matter/cortex. (Assuming that the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) a edema component).

> Two Thirds Between Two Thirds and One

Third Less than One Third Minimal None

Sept 2013 VASARI Feature SetF6 Proportion nCET When scanning through the entire tumor volume, what

proportion of the entire tumor is estimated to represent nonenhancing tumor or nCET (not edema)? Non-enhancing tumor (nCET) is defined as regions of T2W intermediate hyperintensity (less than the intensity of cerebrospinal fluid or vasogenic edema, with corresponding T1W hypointensity) that are associated with mass effect and architectural distortion, including blurring of the gray-white interface. (This may be difficult to discern from vasogenic edema – if uncertain error conservatively). (Assuming that the the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) a edema component.)



F14 Proportion ofEdema

Visually, when scanning through the entire tumor volume, what proportion of the entire abnormality is estimated to represent vasogenic edema? Edema should be greater in signal than nCET and somewhat lower in signal than CSF on T2W or T2W FLAIR. Pseudopods are characteristic of edema extending up to the subcortical white matter but not infiltrating gray matter/cortex. (Assuming that the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) a edema component).



Sept 2013 VASARI Feature SetF5 ProportionEnhancing:

When scanning through the entire tumor volume, what proportion of the entire tumor is estimated to be enhancing. (Assuming that the entire abnormality may be comprised of:(1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) an edema component.)

> Two Thirds Between Two Thirds and One Third Less than One Third Minimal None

F7 ProportionNecrosis

Visually, when scanning through the entire tumor volume, what proportion of the tumor is estimated to represent necrosis. Necrosis is defined as a region within the tumor that does not enhance, is hyperintense on T2W and proton density images, is hypointense on T1W images, and has an irregular border). (Assuming that the entire abnormality may be comprised of: (1) an enhancing component, (2) a non-enhancing component, (3) a necrotic component and (4) a edema component.)

> Two Thirds Between Two Thirds and One Third Less than One Third Minimal None

F4 EnhancementQuality:

Qualitative degree of contrast enhancement is defined as havingall or portions of the tumor that demonstrate significantly higher signal on the postcontrast T1W images compared to precontrast T1W images. Mild = when barely discernible but unequivocal degree of enhancement is present relative to pre-contrast images. Marked = obvious tissue enhancement. (If it does not appear that contrast was administered, select No Contrast Injected.)

No Contrast Enhancement Mild Marked

Sept 2013 VASARI Feature SetF11 Thickness ofenhancingmargin

If most of the enhancing rim is thin, regular, and measures < 3mm in thickness and has homogenous enhancement the grade is minimal. If most of the rim demonstrates nodular and/or thick enhancement measuring 3mm or more, the grade is thick/nodular. If there is only solid enhancement and no rim, the grade is solid. (If it does not appear that contrast was administered, select No Contrast Injected.)

No Contrast Enhancement Minimal Thick/nodular (=> 3mm) Solid

F12 Definition of theEnhancing margin

Assess if most of the outside margin of the enhancement is well defined (i.e. sharply marginated) or poorly-defined (fluffy or indistinct). Are you able to easily trace the margin of enhancement?

No Contrast Enhancement Completely well-defined

(100%) Mostly well-defined (> two

thirds) Mixed (@ 50-50) Mostly poorly-defined (> two

thirds) Completely ill-defined (100%)


F23 Enhancing tumorCrosses Midline:

Enhancing tissue crosses midline is defined by any CET that extends into the contralateral hemisphere through white matter commissures usually expected at the midline (exclusive of herniated ipsilateral tissue).

No Contrast Enhancement No Yes

F31 Heterogeneity When assessing the complexity of the internal architecture of the tumor on FLAIR or T2WI overall grade the uniformity of the tumor matrix (exclusive of what appears to clearly be edema). Select completely homogeneous when the tumor matrix is completely uniform in consistency; mostly homogeneous when more than 2/3 of the tumor matrix is uniform; mixed when about half of the tumor is homogeneous; mostly heterogeneous when more than half of the tumor volume is non-uniform; completely heterogeneous when all of the tumor matrix is non-uniform. FLAIR should be used primarily. T2WI can be used when FLAIR images are not included.

Completely homogeneous (100%)

Mostly homogeneous (> two thirds)

Mixed (@ 50-50) Mostly heterogeneous (> two

thirds) Completely heterogeneous

(100%) Indeterminate


F32 Shape Shape is defined as the overall contour of the abnormal tissue that you would characterize as a tumor boundary. Which descriptor best defines the shape of the entire mass: round, ovoid, lobulated or irregular. Consider the overall boundaries of the NCET and CET exclusive of any edema.

Round/circular/spherical Ovoid Lobulated Irregular None apply

F8 Cyst(s) Cysts are well defined, rounded,often eccentric regions of very bright T2W signal and low T1W signal essentially matching CSF signal intensity, with very thin, regular, smooth, nonenhancing or regularly enhancing walls, possibly with thin, regular, internal septations. Differentiate from a necrotic enhancing tumor cavity with thick irregular walls and complex internal fluid.

Absent Present

Sept 2013 VASARI Feature SetF9 Multifocal orMulticentric

Multifocal is defined as having at least one region of tumor, either enhancing or nonenhancing, which is not contiguous with the dominant lesion and is outside the region of signal abnormality (edema) surrounding the dominant mass.This can be defined as those resulting from dissemination or growth by an established route, spread via commissural or other pathways, or via CSF channels or local metastases, whereas Multicentric are widely separated lesions in different lobes or different hemispheres that cannot be attributed to one of the previously mentioned pathways. Gliomatosis refers to generalized neoplastic transformation of the white matter of most of a hemisphere.

Focal Multifocal or Multicentric Gliomatosis Cerebri

F10 T1/FLAIRRATIO

T1/FLAIR ratio is a gross comparison in the overall lesion size between pre-contrast T1 and FLAIR (in the same plane). Select T1~FLAIR when pre-contrast T1 abnormality (exclusive of signal intensity) approximates size of FLAIR abnormality; Select T1<FLAIR when the size of T1 abnormality is moderately smaller than the surrounding FLAIR envelope; or select T1<<FLAIR when the size of the pre-contrast T1 abnormality is much smaller than size of FLAIR abnormality. (If no FLAIR images were provided select No FLAIR images).

T1~FLAIR T1<FLAIR T1<<FLAIR

Sept 2013 VASARI Feature SetF10 T1/FLAIRRATIO

T1/FLAIR ratio is a gross comparison in the overall lesion size between pre-contrast T1 and FLAIR (in the same plane). Select T1~FLAIR when pre-contrast T1 abnormality (exclusive of signal intensity) approximates size of FLAIR abnormality; Select T1<FLAIR when the size of T1 abnormality is moderately smaller than the surrounding FLAIR envelope; or select T1<<FLAIR when the size of the pre-contrast T1 abnormality is much smaller than size of FLAIR abnormality. (If no FLAIR images were provided select No FLAIR images).

T1~FLAIR T1<FLAIR T1<<FLAIR

F16 Hemorrhage: Intrinsic hemorrhage anywhere in the tumor matrix. Any intrinsic foci of low signal on T2WI (or gradient echo) or high signal on T1WI. Proportion is not a discriminating factor. Select cannot determine if findings are indistinct or may actually represent mineral instead of hemorrhage.

No Yes Indeterminate

F17 Diffusion: Proportion of CET and nCET demonstrating ADC below or same as the ADC of normal-appearing brain. The remainder of the abnormality is assumed to demonstrate increased ADC relative to normal brain. (Based on ADC map only). Select indeterminate when findings are equivocal, (Select no ADC images if ADC images were not provided)

> 2/3 1/3 – 2/3 < 1/3 Minimal (< 5%) None Indeterminate No ADC images


F18 Leptomeningeal invasion:

Leptomeningeal invasion is defined by enhancement of the overlying pia/arachnoid in continuity with enhancing or non-enhancing tumor

Absent Present

F19 Ependymalinvasion:

Ependymal invasion is defined by tumor abutting any adjacent ependymal surface in continuity with enhancing or non-enhancing tumor matrix.

Absent Present

F20 Corticalinvolvement

Cortical involvement is defined by non-enhancing or enhancing tumor that extends to the cortical mantle, or if the cortex is no longer distinguishable relative to tumor.

Absent Present

F21 Deep WMinvasion

Deep white matter invasion is defined by enhancing or nCET tumor extending into the internal capsule, corpus callosum or brainstem. (multiple choices allowed)

None Corpus Callosum Internal Capsule Brainstem


F24 Satellites: A satellite lesion is defined by one or more areas of nCET or CET within the region of signal abnormality surrounding the dominant lesion but not contiguous in any part with the major tumor mass. This is in distinction from a multifocal lesion.

Absent Present

F25 Calvarialremodeling:

Calvarial remodeling is defined as visible erosion/remodeling of inner table of skull (possibly a secondary sign of slow growth).

Absent Present

F29 & F30 Lesion Size Lesion size is defined as the largest perpendicular (x-y) cross-

sectional diameter of entire T2 signal abnormality (longest dimension x perpendicular dimension) measured the single axial image that reveals the largest cross-sectional area of the lesion. (Measurement should incorporate all cardinal imaging features of CET, nCET, necrosis and edema.)

(measured with ruler tool on workstation)

TCGA Glioma Subgroup

Is a geographically distributed collaborative radiogenomic research group comprising multiple individuals from multiple institutions across the US

The Cancer Imaging Archives data samples including imaging and clinical information submitted from participating institutions.

Reason to Update

VASARI featureset was used for high-grade glioma work

For the TCGA group each case was scored by multiple readers; not all readers scored all cases

Interreader reliability analysis was performed for the case-reading rounds for the high-grade tumors

High-Grade Interreader Reliability Analysis

Performed by group member Erich Huang• Categorical features were analyzed with Krippendorff’s α

– −1 ≤ α ≤ 1, with:• α = 1 → perfect agreement• α = 0 → any agreement solely attributed to chance• α < 0 → disagreements are systematic

– Extension of Fleiss’ κ to accommodate the study design where the imaging data for each patient are reviewed by a subset of a pool of readers, but the specific readers that review the imaging data differ for each patient

– Can handle binary and multinomial codings, as well as ordinal

Example Features with High Agreement Were Not Changed

Darker bars = more readers with same score; courtesy Erich Huang

Example Features with High Agreement Were Not Changed


Sometimes Imbalanced Scores Can Lead to Low-Appearing α Despite Visually Apparent Agreement


Reliability of Quantitative FeaturesQuantitative features: used a reliability ratio ρ to assess the inter-reader reliability

ρ = Between-Patient Measurement Variability / Total Measurement Variability

Total Measurement Variability = Between-Patient Measurement Variability + Within-Patient Measurement Variability.

– 0 ≤ ρ ≤ 1, with: • ρ = 1 → no within-patient measurement variability• ρ = 0 → all measurement variability attributed to within-patient measurement

variability

Analyzed square root of the length

Good Reliability of Major Axis Length

Good Reliability of Minor Axis Length

Updating VASARI for Low-Grade Gliomas

Features with poor agreement evaluated for– Clarity of feature definition– Ease of using feature definition– Clarity of feature name– Scientific evidence for any biological relevance

Updating VASARI for Low-Grade Gliomas

Domain experts within the TCGA Glioma group debated and discussed poorly performing features– 11 features subsequently modified or renamed

• 4 were essentially deleted and replaced

– 1 feature was deleted– Some new features were created

The complete, fully updated feature-set is to be made available in a subsequent TCGA publication

Iterative Testing and Updates of VASARI Features

Testing of new/modified features was performed for validation of interreader reliability

Volunteer readers for the TCIA low-grade glioma cases were all trained in multiple sessions by a single trainer on the updated VASARI feature set

Each reader was then assigned a set of 5 test cases for training purposes

Iterative Testing and Updates of VASARI FeaturesFirst round cases were scored using Annotation and Image Markup (AIM) implementation through ClearCanvas on Amazon’s cloud services

Reader scores were compiled and visually compared for consistency– Categorical scores greater than 1 adjacent selection in

distance were considered discrepant (e.g. mild versus severe)

– Scores in adjacent categories were considered reasonable (e.g. minimal vs mild)

First Round Testing

Areas of disagreement greater than 1 category distance apart (yellow) were highlighted

Many areas of reader discrepancy discovered

Images and VASARI definitions were analyzed to understand differences in scoring

First Round Scoring Differences

Multiple etiologies for scoring differences– ADC maps sometimes difficult to find/hidden in

various series on different patients– Different interpretation of feature definitions

(definition clarification and training needed)– Interpretation based on feature name rather than

definition (feature name clarification and training needed)

– Classic interpretative challenges (e.g. edema/non-contrast-enhancing tumor)

Post-First Round VASARI Updates

Multiple teleconferences for consensus among domain experts within the group for additional changes to VASARI feature set to account for the scoring differences

Number of features added: 0Number of features modified/clarified: 9Number of features deleted: 2

Second Round Testing

Though reader agreement was not perfect, it was greatly improved from the first round

Majority of features had few discrepancies greater than 1 category apart

Second Round Scoring Differences

Primary differences from– Interpretive error– Reasonable differences in opinion– User error (mistakenly clicking an incorrect choice)

Freezing the Feature Set

As second round discrepancies appeared reasonable and marginal improvement in scoring reliability from additional feature modification was probably low, the feature set was frozen for use in the next TCGA project: low-grade gliomas

Conclusion

VASARI feature set is a set of controlled categorical descriptors for MR imaging in human gliomas

It has been tested and modified iteratively for best cross-reader reliability in distributed scoring of imaging datasets

Its biological relevance in radiogenomic research is currently being tested against the TCGA dataset

Part of TCGA Glioma Phenotype Research Group

• Hwang, S.N.1 • Holder, C.A.1 • Clifford, R.J.2 • Huang, E.2 • Hammoud, D. 3 • Raghavan, P.4

• Wintermark, M.9

• Gutman, D.A1

• Moreno, C.1

• Cooper, L.1

• Freymann, J.5

• Kirby, J.5

• Krishnan, A.1

• Dehkharghani, S.1

• Jaffe, C.C.6

• Saltz, J.H.1

• Brat, D.J.1

• Colen, R.R.8

• Rubin, D.L.9

• Jain, R10

• Flanders, A.E. 7

• Erickson, B.J.11

• Chen, J.Y. 12

• Rao, A. 8

• Wolansky, L. 13

1Emory University2National Cancer

Institute3National Institute of

Health4University of Virginia5SAIC-Frederick, Inc.6Boston University7Thomas Jefferson

University8MD Anderson Cancer

Center9Stanford University10Henry Ford Hospital11Mayo Clinic12UCSD13Case Western Reserve

U.

Thanks and apologies to anyone who should be on this slide and isn’t

Documents

Updates to and Cross-Reader Reliability Validation of the VASARI Brain Tumor Feature Set for Distributed Scoring of Low-Grade Brain Tumor Cases in Radiogenomic