A Tri-layer Framework for A Tri-layer Framework for Volume Rendering Volume Rendering

Fei YangFei Yang

OverviewOverview

Reconsidering the coding framework of Reconsidering the coding framework of volume rendering volume rendering

Comply with the nature of volume renderingComply with the nature of volume rendering

Meet the requirements of both algorithm Meet the requirements of both algorithm research and application assemblage.research and application assemblage.

The Nature of Volume RenderingThe Nature of Volume Rendering

Sample-level / Ray-level processingSample-level / Ray-level processing

Volume Rendering:

Data-level processing:

Filter FilterData Data Data ……

CompositionInterpolation& SamplingVolume Data Samples …

Processed Samples

Pixels

Sample-level processing

Demand DrivenDemand Driven

““Demand Driven Pipeline” used in VTK:Demand Driven Pipeline” used in VTK: C++ Object-Oriented styleC++ Object-Oriented style Advantage: Streaming for large dataAdvantage: Streaming for large data Problems:Problems:

Hard to codeHard to code Inconsistent with GPU codingInconsistent with GPU coding InefficientInefficient

Alternative:Alternative: C style function callsC style function calls Static assemblage of components ( compiling level )Static assemblage of components ( compiling level ) Problem: GUI integrationProblem: GUI integration

Research & Application Research & Application RequirementsRequirements

Common requirementsCommon requirements EfficiencyEfficiency

Research requirementsResearch requirements Replaceable componentsReplaceable components Timing for performance studyTiming for performance study CPU/GPU comparisonCPU/GPU comparison

Application requirements:Application requirements: High-level interface for common application assemblage:High-level interface for common application assemblage:

Interactive Volume Rendering, fixed data, variable view, render to screen, Interactive Volume Rendering, fixed data, variable view, render to screen, screen resolutionscreen resolution

Low-level control for special applicationsLow-level control for special applications Time series dataTime series data RecordingRecording Fine resolution renderingFine resolution rendering Virtual endoscopy Virtual endoscopy

Replaceable componentsReplaceable components Data sourceData source

Single scalar dataSingle scalar data Multi-modal dataMulti-modal data Segmented dataSegmented data Analyzed resultAnalyzed result Time series dataTime series data Storage: texture/global memoryStorage: texture/global memory

InterpolationInterpolation Nearest NeighborNearest Neighbor Tri-linear Tri-linear SplineSpline

Ray traversal Ray traversal Equal distance traversalEqual distance traversal 3D-DDA traversal3D-DDA traversal

Composition StylesComposition Styles Alpha blendingAlpha blending MIPMIP IsosurfaceIsosurface X-ray X-ray

Property MappingProperty Mapping 1D-TF1D-TF 2D-TF2D-TF

Tri-layer FrameworkTri-layer Framework

Module layerModule layer Modules: low-level data structures and basic render-Modules: low-level data structures and basic render-

time functions time functions

Frame layerFrame layer Assemble the modules statically into frame renderersAssemble the modules statically into frame renderers Providing low-level controlProviding low-level control

Scene layerScene layer Providing high-level interfaces, including interaction Providing high-level interfaces, including interaction

interfaces.interfaces.

Tri-layer FrameworkTri-layer Framework

Module layer A B C D E F

Frame layer Frame Renderer 1 Frame Renderer 2

Bottom – UpC Style

Top – DownC++

Object Oriented

Abstract Scene Class

3D Scene Class

Implement System Integration

Implement 3D Manipulation

Volume Rendering Scene1

Volume Rendering Scene2

Scene Layer

Module LayerModule Layer Coding Style: C styleCoding Style: C style

Low-level data structuresLow-level data structures Basic render-time functionsBasic render-time functions

Defined as static functions in render-time header filesDefined as static functions in render-time header files Can be assembled by “include”Can be assembled by “include”

ContentContent Low-level data structuresLow-level data structures

Volume dataVolume data Image planeImage plane Parameters during each processing stepParameters during each processing step

Basic render-time functionsBasic render-time functions Data fetchingData fetching Property mappingProperty mapping Ray-traversal and compositionRay-traversal and composition Ray generation (start/end points of each ray) and image plane accessRay generation (start/end points of each ray) and image plane access ……

Module Layer – Coding Style Module Layer – Coding Style Basic render-time functions defined as static in header files Basic render-time functions defined as static in header files

// SingleScalarVolume.core.h// SingleScalarVolume.core.h

static CoherentVolume *Volume;static CoherentVolume *Volume;

static int vol_dims[3];static int vol_dims[3];static float vol_spacings[3];static float vol_spacings[3];static float vol_size[3];static float vol_size[3];

static inline float GetScalarValue(int x,int y,int z)static inline float GetScalarValue(int x,int y,int z){{

x=max(min(x,vol_dims[0]-1),0);x=max(min(x,vol_dims[0]-1),0);y=max(min(y,vol_dims[1]-1),0);y=max(min(y,vol_dims[1]-1),0);z=max(min(z,vol_dims[2]-1),0);z=max(min(z,vol_dims[2]-1),0);return Volume->GetValue(x,y,z);return Volume->GetValue(x,y,z);

}}

…………

Module Layer – Coding Style Module Layer – Coding Style A “prepare” function in each render-time header for data-transfer before renderingA “prepare” function in each render-time header for data-transfer before rendering Optional: A “clear” function in a render-time hearer for data-release after renderingOptional: A “clear” function in a render-time hearer for data-release after rendering

// SingleScalarVolume.core.h// SingleScalarVolume.core.h

…………static bool SingleScalarVolumePrepare(CoherentVolume *volume)static bool SingleScalarVolumePrepare(CoherentVolume *volume){{

if (!volume) return false;if (!volume) return false;Volume=volume;Volume=volume;volume->GetDimensions(vol_dims);volume->GetDimensions(vol_dims);volume->GetSpacings(vol_spacings);volume->GetSpacings(vol_spacings);vol_size[0]=vol_dims[0]*vol_spacings[0];vol_size[0]=vol_dims[0]*vol_spacings[0];vol_size[1]=vol_dims[1]*vol_spacings[1];vol_size[1]=vol_dims[1]*vol_spacings[1];vol_size[2]=vol_dims[2]*vol_spacings[2];vol_size[2]=vol_dims[2]*vol_spacings[2];return true;return true;

}}

Frame LayerFrame Layer Data structure assemblageData structure assemblage

class MIPVRclass MIPVR{{public:public:

CoherentVolume* m_Volume;CoherentVolume* m_Volume;MIPRayCasterParam *m_RayCasterParam;MIPRayCasterParam *m_RayCasterParam;VRSceneParam *m_VRSceneParam;VRSceneParam *m_VRSceneParam;VRImage *m_VRImage;VRImage *m_VRImage;

void Render ();void Render ();};};

An assembled structure for each volume rendererAn assembled structure for each volume renderer Low-level data structures can be shared between CPU and GPU frame renderersLow-level data structures can be shared between CPU and GPU frame renderers

Frame LayerFrame Layer Function assemblage: just “Include” and assemble the basic render-time functionsFunction assemblage: just “Include” and assemble the basic render-time functions

#include "SingleScalarVolume.core.h"#include "SingleScalarVolume.core.h"#include "MIP/MIPRayCaster.core.h"#include "MIP/MIPRayCaster.core.h"#include "TypicalRayProducer.core.h"#include "TypicalRayProducer.core.h"

void MIPVR::Render ()void MIPVR::Render (){{

if (!m_Volume || !m_RayCasterParam || !m_VRSceneParam || !m_VRImage) return;if (!m_Volume || !m_RayCasterParam || !m_VRSceneParam || !m_VRImage) return;

if (!SingleScalarVolumePrepare(m_Volume)) return; if (!SingleScalarVolumePrepare(m_Volume)) return; if (!MIPRayCasterPrepare(m_RayCasterParam)) return;if (!MIPRayCasterPrepare(m_RayCasterParam)) return;

GenerateImage(m_VRImage,m_VRSceneParam); // Entrance in "TypicalRayProducer.core.h"GenerateImage(m_VRImage,m_VRSceneParam); // Entrance in "TypicalRayProducer.core.h"

}}

A Typical Calling-chainA Typical Calling-chain

MIP Volume RenderingMIP Volume Rendering

Ray generation and image plane

access:

Module: “ Typical Ray Producer”

GenerateImage ()For each ray, call CastRay()

Ray-traversal and composition:

Module: “ MIP Ray Caster”

CastRay ()For each sample location, call

GetScalarValue()

Data fetching:Module:

“ Single Scalar Volume”GetScalarValue()

Different CombinationsDifferent Combinations

MIP IsosurfaceAlpha Blending

(Full VR)

Ray generation/ Image plane

access

Typical Ray Producer

Typical Ray Producer

Typical Ray Producer

Ray-traversal and composition

MIP Ray-Caster

IsosurfaceRay-Caster

FullRay-Caster

Propertymapping

\ \TF-1DMapper

Data fetchingSingle Scalar

VolumeSingle Scalar

VolumeSingle Scalar

Volume

Scene LayerScene Layer

A standard object-oriented hierarchyA standard object-oriented hierarchy Abstract scene class: Abstract scene class:

Define a series of interfaces for event handlingDefine a series of interfaces for event handling Can be integrated in various GUI contextsCan be integrated in various GUI contexts

3D scene class:3D scene class: Implement event handling interfaces for basic 3D Implement event handling interfaces for basic 3D

manipulationmanipulation Leave the “render” interface for sub-classesLeave the “render” interface for sub-classes

Concrete scene classes:Concrete scene classes: Implement the “render” interfaceImplement the “render” interface Call one of the frame renderers to render each frameCall one of the frame renderers to render each frame

Meeting the RequirementsMeeting the Requirements

Common requirementsCommon requirements Efficiency:Efficiency: static function calls, establish at combining static function calls, establish at combining

time, no pointers, can be inline.time, no pointers, can be inline.

Research requirementsResearch requirements Replaceable components: Replaceable components:

replaceable components provided at module layer replaceable components provided at module layer Timing for performance studyTiming for performance study

Use the scene layer interface to time each frameUse the scene layer interface to time each frame CPU/GPU comparisonCPU/GPU comparison

One-one corresponding components of CPU and GPU can One-one corresponding components of CPU and GPU can be builtbe built

Some low-level data-structures are shared Some low-level data-structures are shared

Meeting the RequirementsMeeting the Requirements

Application requirements:Application requirements: High-level interface for common application High-level interface for common application

assemblage: assemblage: Use the scene layer interfaceUse the scene layer interface

Low-level control for special applicationsLow-level control for special applications Time series data:Time series data: use the frame layer interface or replace the use the frame layer interface or replace the

data fetching module data fetching module Recording: Recording: access the image plane access the image plane Fine resolution rendering:Fine resolution rendering: use the frame layer / module layer use the frame layer / module layer

interfacesinterfaces Virtual endoscopy:Virtual endoscopy: modify the even handling in the scene modify the even handling in the scene

layerlayer

Implementation and AvailabilityImplementation and Availability

As a part of MITKAs a part of MITK A synthetic medical imaging toolkitA synthetic medical imaging toolkit Libraries available at Libraries available at www.mitk.net Implemented mainly in Implemented mainly in

mitkVisualization2 mitkVisualization2 mitkCUDAVolumeRenderingmitkCUDAVolumeRendering

VolTKVolTK Open-source project at Open-source project at www.fei-yang.org

Different Project ArrangementsDifferent Project Arrangements

MITK:MITK:

mitkVisualization2

CPU Scene Layer classes

CPU Module Layer

CPU Frame Layer

mitkCUDAVolumeRendering

GPU Scene Layer classes

GPU Frame Layer

GPU Module Layer

VolTK:VolTK:

VolTKStandard

CPU Module Layer

CPU Frame Layer

VolTKCUDA

GPU Module Layer

GPU Frame Layer

VolTKScene

GPU Scene Layer

Classes

CPU Scene Layer

Classes

The PhilosophyThe Philosophy

What is Object-Oriented-ProgrammingWhat is Object-Oriented-Programming Binding data-structures with corresponding algorithmsBinding data-structures with corresponding algorithms

Procedures as objectsProcedures as objects Storing the parametersStoring the parameters Shared data-objects needs special treatmentShared data-objects needs special treatment

A procedure related to multiple objectsA procedure related to multiple objects Use procedure programming – more suited, easy to Use procedure programming – more suited, easy to

understand.understand.