AVT Phase II Use Case Model

Siemens 1

AVT Phase II Use Case Model

Bob SchwankeVersion of 2008-Jan-19

Siemens 2

How to read this document• A Use Case is a category of behavior

– Use Cases are organized as a hierarchy, partially ordered by• Includes• Includes, optionally [“is extended by” in UML]• Has special case [“has subclass” in UML]

– Use Case hierarchy has a single root– The “includes” list only suggests an ordering, but does not specify the order or

the number of repetitions– A composite use case usually has 3-9 children

• A Scenario is an example of a Use Case– Specifies a sequencing of parts of the use case

• { braces } contain implementation priorities– {1} will certainly be built for SIIM ’09, to support the first user study– {2} will probably be built for SIIM ’09, to support additional user studies specified

in this model– {later} will probably not be built for SIIM ’09

• Also note Vocabulary and Terms to Avoid

Siemens 3

Vocabulary• Scan: one logical image, whether 2D, 3D, or 4D; includes PET-CT

fused data scan.• Scan-series: a temporal sequence of 2 or more scans of the same

subjectA scan-series is a constructed object that associates two or more scans of

the same subject in a temporal sequence. Scan-series are used for AVT studies that investigate change in tumors from one scan to the next, such as “coffee break” and “before and after treatment”.

• Landmark: a naturally-occurring anatomical feature used for systematic, uniform orientation and measurement.

• Fiducial: an artificial mark, such as a tattoo, placed in or on a subject to facilitate systematic, uniform orientation and measurement.

• Nodule: lesion, tumor, lump, etc.See also Terms to AvoidNote: this terminology is temporary, pending reconciliation with the RadLex

team.

Siemens 4

Terms to AvoidThe following terms are ambiguous and should be avoided.

• Image: always modify it, e.g. DICOM Image or logical image or image processing, or use scan.

• Study: always modify it, e.g. DICOM Study, AVT Study, or reader study.

• Series: always modify it, e.g. DICOM Series or scan-series.

Siemens 5

AVT contextInterfaces:• DICOM Database

– Exchange DICOM objects with other DICOM databases (which are (a) local or (b) caGRID DICOM Data Services) {1}

• AIM Database– Exchange AIM objects with other AIM databases (which are (a) local or (b) caGRID AIM

Data Services). (Data model will be an extension of the AIM object model.) {1}

• AVT Study Database– Exchange AVT study data with other study databases, (a) directly, and (b) eventually using

a suitable caGRID Data Service. {later}

• Manual annotation interface {1}

• Algorithm execution interface– Batch execution of algorithm on many images {2}

• Measurement Variability Analysis Interface {1}

• Installation and configuration interface {1}

• Development and customization interface {1}

• Data management interface– Collect and organize images, annotations, and study data {1}

Siemens 6

AVT {1}

Includes• install AVT {1}

• design AVT study {1}

• (optionally) install image processing algorithm {2}

• (optionally) customize annotation tool {2}

• (optionally) customize measurement variability tool{2}

• (optionally) exchange data with other sites {1}

• conduct AVT study {1}

Context: AVT context

Siemens 7

AVT study (data type) {1}

Includes• Relationship to nominal ground truth: [ unknown | known {1} |

calculating]• General Goal: [ exploration {1} | validation {1} | algorithm improvement]• annotation task to be analyzed {1}

• Source of scan data {1}

• independent variables {1}

• comparisons {1}

• statistical methods {1}

• study representation {1}

Has special case• Example AVT Studies {1}

Part of AVT conduct AVT study algorithmically annotate scans

Siemens 8

annotation task {1}

“Annotation” includes both attaching graphics (markup) to images and attaching structured data (annotations) to scans and scan-series.

Most annotation tasks could be carried out manually, semi-automatically, or automatically. Except where indicated, the model describes all three possibilities with the same use-cases.

Includes, optionally• (manually) read annotation instructions {2}

• scan annotation task {1}

• scan-series annotation task {later}

• evaluate task output: confidence, accuracy {2}

• add audit trail information {2}

• add pedigree information {2?}

Part of AVT AVT studyPart of AVT conduct AVT study algorithmically annotate scans

Siemens 9

scan annotation task {1}

Includes, optionally,• locate task-object {2?}

• measure diameter(s): RECIST {1} , WHO {2?}, Wolfe

• segment task-object {1}

• describe task-object {2?}

• create atlas of scan {later}

• rate scan quality {later}

has special case• annotate scout scan {later}

• annotate tumor {1}

Part of AVT AVT study annotation task

Siemens 10

locate task-object {2?}

Includes locating one or more of the following • Seed point• Maximal slice• Center of mass• Landmark point• Region of interest (ROI) surrounding task

objectPart of AVT AVT study annotation task

scan annotation taskNote: finding an ROI and then segmenting the object in it

can also be called unsupervised segmentation

Siemens 11

task object {1}

Has special case• Nodule {1}

• Lymph node {later}

• Organ {later}

• Landmark {later}

Some of these task object types are not relevant for certain tasks.Includes (attributes)• ???Part of AVT AVT study annotation task scan annotation task

locate task object segment task object describe task object

Part of AVT AVT study annotation task scan-series annotation task track task-object across scan-series

Siemens 12

segment task object {1}

Includes, optionally• mark a seed point contained within the task object {1}

• mark a “stroke” contained within the task object• mark an ROI containing the task-object {later} • algorithmically estimate boundary {1}

• interactively delineate or improve boundaries of entity {1}

• algorithmically calculate diameter(s), area(s) and/or volume {1}

Has special case• 2D segmentation {later} • 2½D segmentation {2?}

• 3D segmentation {1?}

– Done semi-automatically, with nudging to improve on algorithmic estimate– create 3D model out of tumor boundary on each slice or three orthogonal slices

automatic or interactive (nudging)

– Region growing, watershed methods, level sets, etc.Part of AVT AVT study annotation task scan annotation task annotate tumor

Siemens 13

describe task object {2?}

Includes, optionally• Classify• Characterize

– Density– Margin distinctness– Shape regularity– etc.

• Actionability• Attach clinical information to task object

– biopsy info– outcome– etc.

Part of AVT AVT study annotation task scan annotation taskRECIST demo of this capability could perhaps be ready for SIIM 09

Siemens 14

annotate scout scan (localizer) {later}

Purpose: define range and orientation of a diagnostic scan

Includes• (optional) select best slices• Attach landmark points to landmarks

[according to instructions]Special case of AVT AVT study

annotation task scan annotation task

Siemens 15

annotate tumor {1}

Includes, optionally• find tumor (identification, seed point?) {later?} • find center of mass of tumor (algorithmic only) {later?} • find maximal slice (axial slice in which tumor has maximum extension) {later?} • find most aggressive part {later}

• measure longest diameter within slice (RECIST) {1}

• measure longest orthogonal diameters (2D for WHO {2?}, 3D for Wolfe Criteria {later} )

• imprecise localization (e.g. ellipse, ROI) {later} • segment task object {1}

• Describe task object (tumor) {later?}

– e.g. density, margin distinctness, attach biopsy information to tumor (biopsy info may have been collected separately under curation)

Note: annotation methods are different depending on the type of tumor, the organ where it is located, and its size, among other things.

Special case of AVT AVT study annotation task scan annotation task

Siemens 16

scan-series annotation task {later}

Includes, optionally

• register scan-series• track task-object across scan-series• Measure change in task-object

– classify change– characterize change– attach additional treatment history (e.g. radiation

therapy)Has scenario

• Oncocare registration scenarioPart of AVT AVT study annotation task

Siemens 17

register scan-series {later}

Includes, optionally• Select ROI to register• Use registration algorithms to estimate registration• Manually adjust registration• 4D registration to account for tumor movement while breathing• measure quality of registration (e.g. normalized mutual information)• visualize quality of registration (e.g. subtraction image)Has special case• Register images of same type (e.g. helical CT)• Register images of different types (e.g. PET/CT, MR/CT,

Helical/Cone CT)Part of AVT AVT study annotation task scan-series annotation task

Siemens 18

registration algorithms {later}

These are automatic registration methods, which the reader may invoke, but which may cause bias if used to speed up defining registration nominal ground truth.

Includes, optionally• Rigid registration• Affine registration• Deformable registrationPart of AVT AVT study annotation task scan-series

annotation task register scan-series

Siemens 19

track task-object across scan-series {later}

Linking objects (e.g. tumors) in two or more scans to indicate that they are believed to be the same object, or one birthed from the other, etc.

Includes, optionally• Linking objects by seed points• Linking objects by centers of mass• Linking objects by contours• Linking objects by maximal slices• Tracking 3D object in 4D scanPart of AVT AVT study annotation task

scan-series annotation taskRECIST Adjudicator could illustrate this use-case{2?}

Siemens 20

OncoCare registration scenario {later}

With two scans,1. Perform rigid registration of whole scans2. Select small ROI around tumor in one scan3. Calculate deformation field to other scan4. Map center of mass of tumor from first to

second scanExample of AVT AVT study annotation task scan-series annotation taskCan be semi-automatic (for manual annotation) or automatic (for auto-

annotate), depending on whether ROI can be found automatically.

Siemens 21

audit trail information {2} The term pedigree, as it appears in the Gap Analysis Document, combines two

concepts: audit trail information and expert commentary, the latter explaining why an annotation is what it is. In this model, we use “pedigree” to refer only to expert commentary. AVT will take care of recording audit trail information automatically, so it is not explicitly listed as a part of any particular use case.

audit trail information includes, optionally• start/source data• changes/measurements made• tools used, with versions• user• date/time of annotationPart of AVT AVT study annotation task

Audit trail information might be attached to almost any annotation.

Siemens 22

pedigree information {2?}

includes, optionally

• methods used to establish nominal ground truth

• subjective uncertainty of observationsPart of AVT AVT study annotation task Part of AVT conduct AVT study curate collection

The term pedigree in the Gap Analysis document combines two concepts: audit trail information and expert commentary, the latter explaining why an annotation is what it is. In this model, we use “pedigree” to refer only to expert commentary.

Pedigree might be attached to almost any annotation. Need better word.

Siemens 23

independent variables {1}

• Scanner variables – Modality– Hardware type (field strength, detector number, etc.)– Slice thickness, spacing, exposure

• Subject variables (Age, Sex; phantom; species; vitality; country of scan; organ scanned; time between scans in scan-series)

• Disease variables (e.g. Referral criteria; outcomes)• Reader variables (incl. inter-reader variation)• Algorithm variables• repetition method (incl. intra-reader variation)Part of AVT AVT study

Siemens 24

scan (data type) {1}

Has special case• Modality-type scan, e.g.

– CT scan– MR scan– PET scan– PET-CT (fused data) scan– Nuclear scan– Ultrasound (3D, 4D) scan– Xray (e.g. mammogram)– Etc. ?

• Organ-type scan, e.g.– Liver scan– Lung scan– Brain scan– Breast scan– Etc.

• Subject-type scan, e.g.– Patient scan– Healthy subject scan– Animal scan– Phantom scan– Post-mortem scan?

• Derived scan, e.g.– Translated/rotated scan– Injected-noise scan– Simulated scan

• (derived from statistical characteristics of real scans? derived from examples?)

– Manipulated scan• Includes hybrids, such as simulated tumors added

to images of healthy patients.Part of AVT AVT study independent variables Part of AVT conduct AVT study curate collection

Scan: one logical image, whether 2D, 3D, or 4D.The special cases below are only intended to note some of the DICOM attributes that are relevant to

AVT scans.Each special case requires certain parametric differences in how annotation tasks are performed.

Siemens 25

repetition method {1}

Has special case

• Rescan before disease progresses (e.g. minutes, days)

• Re-reading same image after “forgetting”• Reading derived imagesPart of AVT AVT study independent variables

Siemens 26

comparisons {1}

Includes• attributes to compare• data organization for inputs and outputs of comparisonsIncludes, optionally• comparisons to nominal ground truth annotations {1}

• diameter comparisons {1}

• volume comparisons {1}

• registration comparisons {later}

• alignment comparisons {later}

• description comparisons {later}

• More, TBDPart of AVT AVT studyPart of AVT conduct AVT study compare annotations

• ? Reqt: Job batching

Siemens 27

diameter comparisons {1}


• RECIST difference (1D) {1}

• WHO difference (2D) {2?}

• Wolfe difference (3D) {later}

• normalized RECIST difference {1}

• normalized WHO difference {2?}

• Normalized Wolfe difference {later}

Part of AVT AVT study comparisonsPart of AVT conduct AVT study compare annotations comparisons

Siemens 28

volume comparisons {1}

Includes, optionally• volume difference {1}

• abs. of volume difference {1}

• normalized volume difference {1}

• abs. of normalized volume difference {1}

• volume overlap ratio {later}

• over-segmented ratio {later} (what is this???) • under-segmentation ratio {later}

• surface distance (RMS, 75th percentile, maximum) {later}

• smoothness difference {later}

• shape difference {later}

• connectedness difference {later}

Part of AVT AVT study comparisonsPart of AVT conduct AVT study compare annotations comparisons

Siemens 29

statistical methods {1}

Includes, optionally• mean {1}

• Bias {1}

• SD (Standard Deviation) {1}

• CV (Coefficient of Variation) {later}

• linear regression {later}

• correlation {later}

• Kruskal-Wallis analysis• multiple comparison {later}

• Wilcoxon signed rank test {later}

• sensitivity, specificity {later}

• PPV (positive predictive value), NPV (negative predictive value {later}

• STAPLE (boundary or classification agreement metric, pixel by pixel???) {later}

• ROC (Receiver Operating Characteristic) curve (efficiency of classification) {later}

• FROC(efficiency for detecting targets) {later}

• Bland-Altmann charts {1}

• Box-and-Whisker charts {1}

• ANOVA methods {1}

• outlier criteria {1}

• More, TBD

Part of AVT AVT studyPart of AVT conduct AVT study

analyze annotation variability

Siemens 30

outlier criteria {1}

Includes, optionally• Outlying data points {1}

– Scaling by interquartile range (IQR) {1}

– Scaling by standard deviations (SD) {1}

– Cutoffs (e.g. 25th percentile, ±2 SD) {1}

• Outlying sub-distributions (means and SDs) {2}

– Scaling by SD of means {2} – Scaling w.r.t. Levene test? {later?} – Scaling w.r.t. Bartlett test? {later?} – Scaling w.r.t. ANOVA test? {2?} – Scaling w.r.t. Kruskal-Wallis test? {later?}

Part of AVT AVT study statistical methodsA sub-distribution is a distribution of values of a dependent variable drawn from a subset

of the data, (or, equivalently, a sub-group of the cases), typically selected by a query that restricts the ranges of the independent variables, e.g. “just the women over age 65”. If a sub-distribution has significantly different aggregate statistics from the remainder of the collection, it may indicate a source of variation.

Siemens 31

study representation {1}

These are AVT-specific detailsIncludes• Annotation labeling plan {1}

– Need labels on annotations indicating their roles in the study, so that AVT can find and process them.

• Measurement and statistics representation (“data cube” schema) {1}

– Specifies how the independent and dependent variables, comparisons, and statistical calculations are organized.

• Data views (e.g. image views, table views, chart views) {1}

– How the data appears in the user interface of AVT’s MVT.Part of AVT AVT study

Siemens 32

example AVT studies {1}

Has special case

• RadPharm Study{2?}

• CDRH Phantom Tumor Reader Study{1?}

• RIDER-Based Study• UCLA Extended Coffee Break Study• DCE-MRI Study of Glioblastoma Multiforme (GBM) • AVW Platform Liver Tumor Study{?}

• U. Maryland Injected Noise Study{?}

• More, TBD• Template for Example AVT StudyIs special case of AVT AVT study

Siemens 33

RadPharm Study {2?} (draft)

Context: measuring tumor change in drug clinical trialsRelationship to nominal ground truth: calculate, then

knownGoal: validate a tumor segmentation algorithmAnnotation task: annotate tumor, esp.• measure diameters: RECIST, WHO• segment tumor, esp.

– 3D segmentation, volumeSource of scan data• MICCAI Data?• FDA Phantom Tumor Data?Independent variables:• Scanner variables: TBD• Subject variables: TBD• Disease variables: TBD• Reader variables

– N human readers– nominal ground truth Annotations performed in AVT– Inter-reader variation

• Algorithm variables– Semi-automatic segmentation performed off-line and

transcoded from DICOM SR to AIM• repetition method: intra-reader variation, more TBD• Fallback plan: If semi-automatic algorithm not

available in time, just study inter-reader variability

Comparisons, esp.• comparisons to nominal ground truth annotations• volume comparisons, esp.

– volume difference– abs. of volume difference– normalized volume difference– abs. of normalized volume difference– Not interested in volume overlap measurements

• diameter comparisons , esp– RECIST difference (1D)– normalized RECIST difference

Statistical methods, esp.• Mean• bias• SD (Standard Deviation)• Bland-Altmann Plot• ANOVA methodsstudy representation: TBD

Special case of AVT AVT study example AVT studies

Siemens 34

CDRH Phantom Tumor Reader Study {1?}

(very early draft)Context: analyzing sources of variation in tumor

imagingnominal ground truth: knownGeneral Goal: exploration of reader variationannotation task: annotate tumor, esp.• manual RECIST and WHO annotations• semi-automatic volume segmentationSource of scan data: FDA Phantom Thorax

nodule dataindependent variables• CT Acquisition parameters: Exposure, Pitch,

Collimation, Slice Thickness, Reconstruction Kernel• Scanner variables: CT: Exposure, Pitch,

Collimation, Slice Thickness, Reconstruction Kernel• Subject variables: Phantom; thorax• Disease variables: (Nodule characteristics:

Attached, unattached, spherical, elliptical, lobulated, spiculated, random)

• Reader variables: Human– Using AVT annotation tool– Inter-reader variation– Etc, TBD

• Algorithm variables: TBD• repetition method: repositioning, intra-reader

variation

Comparisons• comparisons to nominal ground truth

annotations– RECIST and WHO diameters and volume known by

construction of phantom. • diameter comparisons , esp. RECIST, WHO• volume comparisons, esp.

– Tumor 3D segmentation and its volume– Interested in volume overlap measurements?– How do(es) algorithm(s) compare to human readers?

statistical methods• Bias• Mean error• variance• Prioritize outliers by measuring in multiples of

Standard Deviation or Inter-quartile range• Box-and-whiskers plots• Bland-Altmann Plot?• See Chenyang’s team’s papers?study representation: TBD


Siemens 35

RIDER-Based Study(draft)

Context: measuring tumor change in patient image datasetsGround truth: not knownGoal: Evaluate methods to estimate change in tumor volume

(segmentation, calculation methods).Annotation task: annotate tumor, esp.• measure diameters: RECIST, WHO• segment tumor, esp.

– 3D segmentation, volumeSource of scan data• Thin Slice data from RIDER study (or other similar)• Thoracic CTs of patients with known lung lesions (primary or mets)• At least one followup study (to assess change over time)Independent variables:• Scanner variables: Thin slice (thickness <= 1.5 mm)• Subject variables: Must have multiple patient scans performed

at different times (to assess change)• Disease variables: Lung Cancer primary or disease that is

metastatic to the lung– Prefer to have lesions of different sizes– Perhaps have lesions of different shapes

• Reader variables– N human readers– Ground Truth Annotations performed in AVT


transcoded from DICOM SR to AIM• Identify same lesion at multiple time points to assess change• repetition method: none• Fallback plan: If semi-automatic algorithm not available in time,

just study inter-reader variability

Comparisons, esp.• For each lesion, calculate change over time

– volume comparisons, esp.• volume difference• abs. of volume difference• normalized volume difference• abs. of normalized volume difference• Not interested in volume overlap measurements• May be interested in categorical variable such as

regression, progression or stable– diameter comparisons , esp

• RECIST difference (1D)• normalized RECIST difference

• Intra-observer variation (details?)• Inter-observer variation (details)• Subanalyses

– Effect of tumor size (< 1 cm diam vs. > 1 cm diam)– Effect of tumor shape (e.g. irregular vs. spherical?)

Statistical methods, esp.• Mean change (for each method)• Bias (for each method)• SD (Standard Deviation) within a method• ANOVA methodsstudy representation: TBD


Siemens 36

UCLA Extended Coffee Break Study (draft)

Context: measuring tumor change in patient image datasets under coffee break paradigm (no change condition)

Ground truth: not known, but should be no change conditionGoal: Evaluate methods to estimate change in tumor volume

(segmentation, calculation methods); establish methods that minimize change; establish minimum detectable change.

Annotation task: annotate tumor, esp.• measure diameters: RECIST, WHO• segment tumor, esp.

– 3D segmentation, volumeSource of scan data• MSK coffee break experiment data• Thoracic CTs of patients with known lung lesions (primary or mets)• 32 pairs of thoracic CT scans • Performed on same patient within 15 min interval on same scanner

with same technical settings.Independent variables:• Scanner variables: All technical parameters maintained the

same for each pair of scans.• Subject variables: each will have two scans performed over a

period of ~ 15 minutes• Disease variables: Lung Cancer primary or disease that is

metastatic to the lung• Reader variables

– N human readers– Ground Truth Annotations performed in AVT


transcoded from DICOM SR to AIM• Identify same lesion at each time point to assess change• repetition method: none• Fallback plan: If semi-automatic algorithm not available in time,


a.k.a. QIBA Group 1B Patient datasets Extended Coffee Break Study

Comparisons, esp.• For each lesion, calculate volume

– volume comparisons, esp.• volume difference• abs. of volume difference• normalized volume difference• abs. of normalized volume difference• Not interested in volume overlap measurements• May be interested in categorical variable such

as regression, progression or stable– diameter comparisons , esp


• Intra-observer variation (details?)• Inter-observer variation (details)Statistical methods, esp.• Mean change across lesions (for each method)• Bias (for each method)• SD (Standard Deviation) within a method• ANOVA methodsstudy representation: TBD


Siemens 37

DCE-MRI Study of Glioblastoma Multiforme (GBM)

Context: measuring change in GBM tumors to predict response in clinical trial

Ground truth:clinical outcomeGoal: Evaluate methods to estimate change in tumor function using

DCE-MRI image acquisition and derived parameters (segmentation, calculation methods). Compare response to standard measures using RECIST, WHO.

Annotation task: annotate tumor, esp.• measure diameters: RECIST, WHO• segment tumor, esp.

– 3D segmentation, volumeSource of scan data• DCE-MRI data of Patients with GBM• Repeat studies as long as patient is on trialIndependent variables:• Scanner variables: Performed as part of a clinlcal trial; some

control of acquisition parameters, but not completely specifiied.

• Subject variables: each patient has GBM; many will have multiple scans over treatment time frame

• Disease variables: GBM• Reader variables

– N human readers– Ground Truth Annotations performed in AVT


transcoded from DICOM SR to AIM• Identify same lesion at each time point to assess change• repetition method: none• Fallback plan: If semi-automatic algorithm not available in time,


Comparisons, esp.• For each lesion, calculate functional parameters such as

perfusion• Calculate on a voxel by voxel basis• Comparison to contralateral side or non-tumor region• Change in perfusion before and after treatment• Compare to linear measurements of tumor size estimated from

MRI– volume comparisons, esp.

• volume difference• abs. of volume difference• normalized volume difference• abs. of normalized volume difference• Not interested in volume overlap measurements• May be interested in categorical variable such

as regression, progression or stable– diameter comparisons , esp


Statistical methods, esp.• Mean change across lesions (for each method)• Bias (for each method)• SD (Standard Deviation) within a method• ANOVA methodsstudy representation: TBD


Siemens 38

AVW Platform Liver Tumor Study

TBDSpecial case of AVT AVT study example AVT studies

Siemens 39

Template for Example AVT StudyContext:nominal ground truth: [ unknown | known | calculating]General Goal: [ exploration | validation | algorithm

improvement]annotation task to be analyzed• scan annotation task• scan-series annotation taskSource of scan dataindependent variables• Scanner variables • Subject variables (Age, Sex; phantom; species; vitality;

country of scan; organ scanned)• Disease variables (e.g. Referral criteria; outcomes)• Reader variables, incl. inter-reader variation• Algorithm variables• repetition method: [Repositioning | Re-reading | derived

images], incl. intra-reader variationComparisons• attributes to compare• comparisons to nominal ground truth annotations• volume comparisons• diameter comparisons • registration comparisons• alignment comparisons• description comparisons

statistical methods• Mean• bias• SD (Standard Deviation)• CV (Coefficient of Variation)• linear regression• correlation• Kruskal-Wallis analysis• multiple comparison• Wilcoxon signed rank test• sensitivity, specificity• PPV (positive predictive value), NPV (negative predictive value• STAPLE (boundary or classification agreement metric, pixel by

pixel???)• ROC (Receiver Operating Characteristic) curve (efficiency of

classification)• FROC(efficiency for detecting targets)• Bland-Altmann charts• Box-and-Whisker charts• ANOVA methods• outlier criteria

study representation: TBD


Copy and modify this template to describe your study. Follow links for relevant parts of business use-case model.

Siemens 40

install image processing algorithm {2}


• install c++ image processing algorithm in a scene graph library

• Install algorithm written in another programming language in a scene graph library

• Configure AVT to load custom scene graph library

• Configure a scene graph pipeline to implement a (new) image processing algorithm

Part of AVT

Siemens 41

install C++ image processing algorithm {2}

Includes

• Embed image processing algorithm in a subclass of a suitable XIP scene graph component class

• Compile and link scene graph component into a binary library (e.g. a DLL in Windows)

Part of AVT install image processing algorithmAVT provides an example to copy and modify.

Siemens 42

customize annotation tool {2}

Tailor the annotation tool to fit the intended AVT study, thereby eliminating extraneous variability caused by using a too-general tool.

Includes, optionally• develop IA user interface components

to fit AVT study• develop IA scene graph to fit AVT

studyPart of AVT

Siemens 43

customize measurement variability tool {2}

Customize MVT to support the particular kinds of calculation and exploration needed for the intended statistical scan. For data exploration, typically means adding new modules to the growing bag of tricks, and perhaps configuring the menus to conveniently access frequently-used ones. For the actual validation, may involve writing new components that pull together chosen analysis methods to present the results.

Includes, optionally• Develop R package to fit MVT/R interface• Load R package into MVT• Configure MVT menus• Configure MVT tabcards or write new ones• Configure MVT report generators or write new ones• Configure MVT table views or write new ones• Configure MVT image views or write new ones• Configure MVT image scene graphPart of AVTAVT supplies an example to copy and modify

Siemens 44

conduct AVT study {1}

Includes, optionally• curate collection {1}

• manually do annotation task specified in AVT study {1}

• algorithmically annotate scans {2}

• compare annotations {1}

• analyze annotation variability {1}

Part of AVT

Siemens 45

curate collection {1} Includes• Define criteria for collecting scans, {1} e.g.

– Age, Sex, Referral criteria, Hardware type (field strength, detector number, etc.)– Slice thickness, exposure– Country of scan, scan subject type

• Define criteria and instructions for annotating scans {1}

– Need automated query support for selecting based on these criteria• Add scans to collection {1}

• (optional) create scan-series and add to collection {later}

• add vetted patient demographics (maybe from DICOM header) {1}

• add vetted clinical data (such as outcomes) {1}

• add pedigree information to whole collection {2}

• certify nominal ground truth {1}

Part of AVT conduct AVT study

Siemens 46

certify nominal ground truth {1}

nominal ground truth is established on scans and scan-series, typically by statistically analyzing multiple attempts to estimate that annotation, manually and/or automatically, and recording the estimate as a new annotation on the relevant case.

nominal ground truth can also be associated with histology, diagnoses, pathology, and follow-up

includes, optionally• Add an annotation to an annotation indicating that it is the most

recent best estimate of the nominal ground truth for that annotation type, for the object being annotated.

• Add a pedigree note to a collection indicating that a certain tag on annotations in that collection marks it as nominal ground truth for a specified purpose. The note also documents how the nominal ground truth was determined.

Part of AVT conduct AVT study curate collectionNeed version tracking information to handle cases where nominal ground truth evolves.

Siemens 47

algorithmically annotate scans {2}

Includes• Specify algorithms to run to perform annotation task

specified in AVT study {2}

• configure labeling scheme for generated annotations, so they can be found later for analysis {2}

• Specify sampling scheme for choosing which scans in collection will be annotated {later}

– Sampling from space of <scans x algorithms x parameters>• Execute specified algorithm on chosen scans {2}

• Review results of algorithm execution (did it do what I meant?){2}


Siemens 48

compare annotations {1}

Includes• Execute comparisons specified in

AVT study• Review comparisons (did it do what

I meant?)Part of AVT conduct AVT study

Siemens 49

analyze annotation variability {1}

Includes• Calculate statistics specified in AVT study

{1}

• review statistics results {1}

• Generate variability reports specified in study representation of AVT study {1}

– E.g. statistics reports with some typical images


Siemens 50

review statistics results {1}

Review data, interactively, as specified in study representation of AVT study.Includes, optionally• Prioritized lists of outliers {1}

• Prioritized lists of outlying sub-distributions {2}

• Graphs {1}

• Tables {1}

• Drill-down to details of outliers– view scans {1}

– sub-distributions {2}

• More, TBDPart of AVT conduct AVT study

analyze annotation variability

Siemens 51

Backup slides

No! Don’t Back Up!

Siemens 52

Concepts not yet covered adequately in model

From UCLA team• Algorithms involving functional measures

(Dynamic Contrast Enhanced MRI, for example)

• Other Oncology examples (talk about later)• Morphological changeSuggestions solicited for how to incorporate

these in model.

Documents

AVT Phase II Use Case Model