Interaction Techniques in Medical Volume Visualization Bernhard Preim

Interaction Techniques in Medical Volume Visualization

Bernhard Preim

Interaction Tasks and Techniques

Interaction Tasks• Exploration of original data• Manipulation of transfer functions• Multiplanar reformatting (MPR)

Bernhard Preim 2

Exploration of Original Data

Bernhard Preim 3

• “Browsing” through the slice data• Simple contrast and brightness

control via mouse movement (windowing also called ramp transfer function)

• Flexible definition of slices in a corresponding visualization

• Cine mode for animation impression

• Opening and closing of a legend in the viewerPatient information (name, date of birth, gender, Id)Image information (modality, voxel size, recording date)Option: more or less detailed legend

• Synchronized display of two data sets Example: Liver CT; first data set without contrast agent, second data set with CASynchronization related to windowing parameters and the displayed layer

• Selection of the viewing direction (coronary, sagittal, axial)

Bernhard Preim 4


Example for legends, data: Univ. Hospital Leipzig

Bernhard Preim 5


Change of contrast and brightness, data: Univ. Hospital Leipzig

Bernhard Preim 6


Moving of a cross line in communicated views (Peter Hastreiter, Uni Erlangen)

Bernhard Preim 7

Historical model:Drawings by Dürer


Transfer functions: Mapping of data onto presentation parameters (colors, gray values, transparency)

• Determine the visibility and perceptibility of structures• Parametrization of TFs is an essential interaction for the

exploration of volume data.

Challenges:• Exploration of data sets with unknown structures• Exploration of data sets with different structures of similar

intensity

Bernhard Preim 8

Transfer Functions

Three volume visualizations of one CT data set with different opacity transfer functions. No exliplicit classification is applied, leading to the problems with „teeth“

Bernhard Preim 9

Skin Bones Teeth

Transfer Functions

Requirements• Selection of predefined TFs (e.g. liver CT, lung CT)• Targeted search for suitable TFs• Correlation between adjustable parameters and characteristics

of the resulting images• Definition flexibility• Fast preview

Bernhard Preim 10

Transfer Functions

Typical transfer functions:• Windowing (ramp)• Bi-/trilevel windowing• Inverse windowing• Piecewise linear functions• Polynoms of higher degree/splines

Problem: No recognizable relation between TF characteristics and visualization

Bernhard Preim 11

Transfer Functions

Thorax CT data set, emphasis of skeletal structures

Bernhard Preim 12

Transfer Functions

Bernhard Preim 13

Thorax CT data set, emphasis of blood vessels

Transfer Functions

Representation and application of TFs• Discrete representation in lookup tables• Size: e.g. 4096 entries with 32 bit (8 bit each for RGB and

alpha)• Hardware support for Lookup tables

Problem: hardware dependency w.r.t. size of color tables

Bernhard Preim 14

Transfer Functions

Transfer Functions

Sophisticated concepts:

• Stochastic generation of TFs that may be selected by the user (multilevel iterative search), presentation as thumbnails (He et al. [1996], König et al. [2001])

• Image-based TF design (Fang et al. [1998])

• Enhanced TF

Integration of image processing filters (e.g. edge recognition)Local TFMultidimensional TF (illustration of derived data, e.g. gradient fields, Levoy [1988])

Bernhard Preim 15

Transfer Functions

Stochastic generation of TFs:

Iterative search process (He et al. [1996]):

1. Use of an initial TF library

2. "Mutation" of this function through a genetic algorithm (25 generations)3. Direct volume rendering (back then with VolVis 100x100 pixel, 10s)4. Subjective result analysis by the user

16Bernhard Preim

17Bernhard Preim

Source:

König et al. [2001]

Transfer Functions

Transfer Functions

Image-based TF design• Idea: Definition of the transfer function, image information

serve as context (Castro et al. [1998])

• Global histogram• Histogram along a layer• Histogram along a ray

Bernhard Preim 18

Transfer Functions

Histogram along the orange ray as context for TF specification

Bernhard Preim 19

Image: Dirk Bartz, Univ. Leipzig

eye ball (light)muscles (dark)

Transfer Functions

RGB Alpha and gray value Alpha TF (Courtesy of Peter Hastreiter, Univ. Erlangen)

Bernhard Preim 20

Transfer Functions

Composition of a TF as weighted sum of component functionsComponents may represent known materials, e.g. fat, bone, …

Parameters of component functions:

Sb, Sc - inner sampling points, Sa, Sd - outer sampling points

Bernhard Preim 21

Transfer Functions

Adaptation of a trapezoid template to the local histogram of a rectangular region.

Bernhard Preim 22

Source: Castro et al. [1998]

The Transfer Function Bake-Off, Data: Sheep heart (IEEE CG&Application 5/6 2001, Pfisterer)• Comparison of different TF specification techniques1. ISO rendering of the segmented raw data (sheep heart) 2. Trial&Error - (20 min) with VolumePro3. Without data model - ISO automatically selected according to the maximum gradient

magnitude4. 2D TF with data model – automatic distance map, semiautom. opacity, manual color map

Bernhard Preim 23

Transfer Functions

Source: Pfisterer [2001]

Transfer Functions

Data-based TechniquesSelection of a transfer function that emphasizes edges.Edge model:

Perfect intersection between 2 structures is "blurred" through an error function (point spread function of the data acquisition). Assumption: Blurring through an isotropic Gaussian function. → fits to CT data well

Source: Kindlman, Durkin [1998]

Bernhard Preim 24

Transfer Functions

Data-based Techniques: edge enhancementEdge criteria: strong gradient g, very small second derivative

h (zero crossing):

-h(v)p(v) =

g(v)

Data values along an edge, 1st and 2nd derivative

Bernhard Preim 25

Transfer FunctionsDetermination of g (v) and h (v) via average determination from all first and second derivatives of all voxels with value v.Internal representation:Histogram volume H:

x-axis → f (v)y-axis → f“(v)z-axis → f´(v)

Algorithm:1. Determine min. and max. valuesfor f‘‘(v) and the maximum for f´(v). Minimum for f´(v) is assumed to be 0.

2. Fill H, whereas the values are scaled such that min and max are depicted from f´ and f´´ to 0 and 256.

26Bernhard Preim

Transfer Functions

Data-based Techniques: edge enhancement• What can be determined from the histogram volume?

Edge positions w.r.t. the data• What can be entered by the user?

• A selection of the "peaks" that shall be depicted• Form of the depicted peaks via boundary emphasis

function (bef)• Typical forms of bef()

Bernhard Preim 27

Transfer Functions

Data-based Techniques: edge enhancement

Applied 2D opacity functionand volume rendering of the Visible Woman data set (TF automatically determined).

The small image indicates the 2DHistogram (intensity values vs. gradient magnitude)

Brightness indicates frequency of the combination.

Source: Kindlmann, Durkin (1998)

Bernhard Preim 28

Transfer Functions

Data-based Techniques: edge enhancementComparison of edge-enhancing direct volume rendering and iso-

surface rendering

Illustration of a spiny dendrite based on microscopy data

Source: Kindlmann and Durkin 1998

Bernhard Preim 29

Transfer Functions

Data-based Techniques: edge enhancement• Preconditions for successful application:

Existence of clear object boundariesHomogenous dataOnly little noise, no "outliers"Medicine: CT data (if CA is applied, it must be equally distributed)

Bernhard Preim 30

Transfer Functions

Data-based Techniques• Use of a once specified TF as reference• Goal: "Re-use" of an empirically specified TF

Application: targeted illustration of a structure in a modality (e.g. aneurysms in MR)Procedure:

Selection of a reference data set Dref and a TF Tref(v)Use of the normalized histograms of the data sets H(Dref )

and H(Dstudy)Non-linear transformation t of the intensity values of Dstudy, such that H (Dstudy) ~ H (t(Dref))Hence, Tstudy (v) = Tref (v)

Bernhard Preim 31

Transfer Functions

Data-based Techniques: reference TF• Determination of the similarity of the histograms

1. Idea: minimization of the histogram distances2. Better idea: use of the p-function by Kindlmann (considers also f‘(v) and f‘‘(v))In case of comparable data sets the p-values are similar to the histograms

Literature: Rezk-Salama et al., VMV [2000]

Bernhard Preim 32

Transfer Functions

Data-based Techniques: Reference TF

• Visualization of blood vessels in the brain with CT angiography, left: no adaptation, middle: illustration of the first idea (histogram transformation), right: adaptation of the p-function Source: Rezk-Salama et al. [2000]

Bernhard Preim 33

Transfer Functions

Multidimensional TFs• 1D TF: Map data onto opacity/colors• Multidimensional TFs: Use additionally derived information,

e.g. strength of the gradient or the second derivative

• Typical example: Adaptation of the opacity to the strength of the gradient, emphasis of data

intersections• Advantage: Additional degrees of freedom to generate

high-quality images• Disadvantage:High interaction costs

Bernhard Preim 34

Transfer Functions

Multidimensional TFs• Consideration of the 2nd derivative (1st derivative of a scalar

field → vector, 2nd derivative → matrix)

• Hessian Matrix:

• Criterion (scalar value) for the 2nd derivative: largest eigenvalue of the Hessian Matrix and strength of the 2nd derivative, respectively in direction to the gradient (instead of the Hessian Matrix)

Bernhard Preim 35

Transfer Functions

Multidimensional TFs• Gradient calculation usually via central differences

• Mapping of the gradient size to the opacity (gradient magnitude weighted transparency)

Bernhard Preim 36

Transfer Functions

Multidimensional TFs• Volume visualization with a

gradient-dependent TF for opacity, according to Levoy [1988]) (Visible Human CT data set)

Bernhard Preim 37

2D Transfer Functions

Starting point for a simple specification: gradient intensity histograms. Filtering is important. Goal: accentuation of intersections.

Bernhard Preim 38

Source: Stölzl [2004]


Bernhard Preim 39

Dense tissue and bone parts with additional gradient emphasis (green marking)Image courtesy: Hoen-Oh Shin and Benjamin King,MH Hannover [2004]


Bernhard Preim 40

More dense soft tissue (yellow marking)Image courtesy: Hoen-Oh Shin and Benjamin King,MH Hannover [2004]


Bernhard Preim 41

Regions with high gradients are visualized (red marking)Image courtesy: Hoen-Oh Shin and Benjamin King, MH Hannover [2004]


Edge detector as input to define arcs

Bernhard Preim 42

Source: Stölzl [2004]

2D Transfer FunctionsLocal TFs• Motivation: Often, global TFs enable no sufficient

differentiation• Another kind of 2D TFs with the 2nd dimension being the

object id• Example: Division of a lookup table into 4 segments for 4

different illustrations

Caution: Interpolation beyond segment borders is not allowed!

Bernhard Preim 43

Source: Rezk-Salama [2002]


Blood vessels in the lung lobes are displayed with separate local TFs.

Bernhard Preim 44


Template-based specification of 2D TFs1D templates:

2D templates:

Bernhard Preim 45

Source: Tappenbeck [2006]

Source: Castro et al. [1998]


Template-based specification of 2D TFs:• Simplification of the interaction is even more important than in

the 1D case• Discretization in a Lookup table• Sufficient size required, at least 256x256• Applicable to arbitrary 2D domains (intensity: gradient

strength, intensity: distance to a target structure, …)

Bernhard Preim 46


Representation of 2D TFs in a rectilinear grid as basis for discretization in an LUT

Bernhard Preim 47


Distance-dependent TFs:• Additional entries:

Segmented target structure (tagged volume)Distance transformation w.r.t. the target

• Use of an editor for 2D TF• Sample applications:

Fade-in of safety margins around tumorsOpacity control in case of large organs

Bernhard Preim 48



Target structure: lung surfaceSelection of interesting structures by intensity and distances

Bernhard Preim 49


Resulting Volume Visualization (quite useless example; but appropriate illustration of the concept). Distance to the lung is used to assign different colors and opacity values.

Bernhard Preim 50



Useful example: Emphasis of blood vessels in certain distances around a tumor via distance-dependent TFs

Bernhard Preim 51



Size-Based Transfer Functions

Bernhard Preim 52

Left: With a 1D TF vessels are emphasized. Right: A size-based TF is adjusted to emphasize larger structures. Intensity and size are combined and yield a clear view to an aneurysm (From: Correa, 2008).

Size-based TFs are designed to emphasize features of a certain size.Basis: (Continuous) multi-scale representation of the dataset defined by a diffusion process or wavelets.

Size-Based Transfer Functions

Bernhard Preim 53

Left: With a 1D TF vessels and bones of the hand cannot be discriminated. Middle: Size is mapped to color to distinguish vessels and bone. Right: A size-based TF is adjusted to emphasize smaller structures making veins visible (From: Correa, 2008).

Size-based TFs were re-used and refined by others, e.g. [Wesarg, 2010]

Stroke-Based Transfer Functions

Bernhard Preim 54

TFs are defined based on user-drawn strokes. The blue stroke indicates what should be displayed and the green stroke what should be omitted (left). The histogramm of the strokes is analyzed (middle) and a TF is defined that separates the regions indicated by the strokes (From: Ropinski, 2008).

Advanced Concepts

Increased GPU support and better performance enables more advanced concepts

Data are analyzed, derived features are used for manipulating the TF

A more goal-directed process is supported with real-time feedback to allow also trial-and-error

Shape-Based Transfer Functions

• Objects are roughly segmented.• Objects are decomposed based on a skeletonization• Segments are merged with sophisticated techniques to

avoid that too small segments are classified• Subobjects are assessed w.r.t.

Tubiness (e.g. vessels, nerves have high values) Surfaceness Blobbyness (e.g. organs have high values)


Subobjects are represented by a point in the barycenter of shapes

A transfer function assigns opacity values based on this shape information

Elongated objects (red dots) are emphasized(From: Praßni, 2009)


Shape-Based TFs may emphasize roughly spherical pathologies, such as polyps in the colon and aneurysms. (The close-ups show renderings with conventional TFs) (From: Praßni, 2009)


Discussion:•Results of shape classification is available via an easy-to-use interface•Related to earlier work from Sato et al. (2000) where similar features are analyzed in a multi-scale approach on a per-voxel basis•No complex histograms need to be analyzed•Combination with size-based TFs would enable more fine-grained control

Visibility-Driven Transfer Functions

A visibility histogramm indicates for each intensity value how visible the voxel‘s with this value are depending on the current viewpoint and the current opacity transfer function

A fast GPU-based histogram computation is essential to update visibility histograms after rotation (Correa and Ma, 2009)

Computation of the visibility histogram(From: Correa and Ma, 2009)

Visibility-Driven Transfer Functions

The intensity histogramme and the visibility resulting from the current TF. Left: The opacity TF makes skin and flesh visibile thus hiding bones despite of the high opacity values (low magenta peak on bones). Right: By chosing low opacity values for skin and flesh they remain noticeable

but now bones are clearly visible (From: Correa and Ma, 2009)

Multiplanar Reformatting

MPR illustration of an MRT data set of the head.Left: The cutting plane indicates which slice is cut out from the original data.

Bernhard Preim 62

Application for vessel diagnostics • MPR is automatically oriented

orthogonal to the vessel centerline • Integrated view of cross section and

3D visualization

Bernhard Preim 63


Local MPR:Rotation around a locally interesting structure (tumor, vessel centerline)

Bernhard Preim 64


Anatomical Reformatting:

Idea: Use of segmentation results for a reformatting during which layers with a constant

distance to an anatomical structure (e.g. an organ surface) arise.

Procedure: - Lines of the data set are shifted against each other in such a way that voxels on the surface are located in a layer vertically to the viewing direction.

- The originally curved slices are viewed in layers → organ-specific coordinate system.

Feature: Anatomically reformatted layers show only voxels with same distances to the organ boundary.

Bernhard Preim 65


Bernhard Preim 66

Anatomical reformatting based on a lung lobe segmentation.Round lesions in the mediastinum.Source: Dicken et al., BVM 2003, Data: Prof. Günther (RWTH Aachen)


Summary

• Suitable interaction techniques are crucial for the practical application of medical visualization techniques.

• On the one hand, the interaction should be simple and clear. On the other hand, it should be flexible enough.

• Presets, or automatically adapting presets, are often a good basis.

• Transfer functions: Image galleries and gradient-dependent TFs are the standard.

Bernhard Preim 67

Literature

S. Castro, A. König, H. Löffelmann, E. Gröller, Transfer Function Specification for the Visualization of Medical Data, Technical Report, Institute of Computer Graphics and Algorithms, Vienna University of Technology, 1998,

C. Correa, K. L. Ma. Size-based transfer functions: A new volume exploration technique. IEEE Transactions on Visualization and Computer Graphics, 14 (6), 1380-1387, 2008

V. Dicken, B. Wein, H. Schubert et al. Projektionsansichten zur Vereinfachung der Diagnose von multiplen Lungenrundherden in CT-Thorax-Aufnahmen, Bildverarbeitung für die Medizin, Springer, Reihe Informatik aktuell, März 2003

S. Fang, T. Biddlecome, and M. Tuceryan. Image-Based Transfer Function Design for Data Exploration in Volume Visualization. In Proc. IEEE Visualization, 1998

T. He, L. Hong, A. Kaufman, and H. Pfister, Generation of Transfer Functions with Stochastic Search Techniques, in Proc. of IEEE Visualization, 1996

G. Kindlmann and J. W. Durkin. Semi-Automatic Generation of Transfer Functions for Direct Volume Rendering, In Proc. of IEEE Symposium on Volume Visualization, 1998

Bernhard Preim 68

LiteratureM. Levoy, Display Surfaces from Volume Data, IEEE Computer Graphics and Applications, 25 (1988)H. Pfister, W. E. Lorensen, C. L. Bajaj, G. L. Kindlmann, W. J. Schroeder, L. Sobierajski Avila, K. Martin,

R. Machiraju, J. Lee: The Transfer Function Bake-Off. IEEE Computer Graphics and Applications 21(3): 16-22 (2001)

J. S. Praßni, T. Ropinski, J. Mensmann, K. Hinrichs. Shape-based transfer functions for volume visualization. Proc. of IEEE PacificVis, 2010, 9-16

C. Rezk-Salama, Peter Hastreiter, J. Scherer, G. Greiner: Automatic adjustment of transfer functions for 3d volume rendering. In Proc. of Vision, Modelling and Visualization, S. 357-364, 2000.

C. Rezk-Salama, Volume Rendering Techniques for General Purpose Graphics Hardware, Dissertation, Philipp-Alexander Universität Erlangen-Nürnberg

T. Ropinski, J. Praßni, F. Steinicke, K. H. Hinrichs: Stroke-Based Transfer Function Design. Proc. of Volume Graphics 2008: 41-48

Y. Sato, C.-F. Westin, A. Bhalerao, S. Nakajima, N. Shiraga, S. Tamura. Tissue classification based on 3D local intensity structures for volume rendering. IEEE Trans. Vis. Graph., 6(2):160–180, 2000.

D. Stölzel. Entwurf gradientenabhängiger 2D-Transferfunktionen für die medizinische Volumenvisualisierung. Master's thesis, Dept. of Computer Science, Univ. MD, 2004.

• Tappenbeck, B. Preim, V. Dicken. Distance-Based Transfer Function Design: Specification Methods and Applications. In Simulation und Visualisierung, pages 259-274, 2006

S. Wesarg, M. Kirschner, M. F. Khan: 2D Histogram based volume visualization: combining intensity and size of anatomical structures. Int. J. Computer Assisted Radiology and Surgery 5(6): 655-666 (2010)

Bernhard Preim 69

Documents

Interaction Techniques in Medical Volume Visualization Bernhard Preim