69
SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer Center University of California, San Diego, USA

SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

Embed Size (px)

Citation preview

Page 1: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Volume GraphicsTechnology to Tools

David R. Nadeau

Principal Scientist

San Diego Supercomputer Center

University of California, San Diego, USA

Page 2: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Technology & Tools

• Technology = a useful idea– Polygons, images, volumes, transforms, shading, …

• Tool = an idea put to use– Visualization, art, story telling, games, …

• As a technology matures, it becomes a tool– We focus more on its use, than its mechanism

– Use really explodes when artists begin to use it

Page 3: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Technology & Tools

• Written storytelling – dramatically advanced by…– Printing press, movable type, word processing, e-mail,

Internet, …

• Music composition – dramatically advanced by…– Standardized notation, equal-tempered tuning, piano,

electric guitar, synthesizer, digital signal processing, …

• Computer graphics – dramatically advanced by…– Hmmm…?

Page 4: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Topics of the Talk

• History– What has happened so far?

• Rendering– What can we do now?– What would we like to do?– What might we be able to do in the future?– How might this affect the way we render volumes?

• Authoring– How might rendering issues affect how we create volumes?– What might future authoring tools look like?

HistoryRenderingAuthoring

Page 5: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

HistoryRenderingAuthoring

A Brief Look atGraphics History…

• What are some major events?• What did they enable?

• Surface vs. Volume graphics

• Technology vs. Story telling vs. Games

HistoryRenderingAuthoring

Very

Page 6: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of surface graphics

• Selected Technology events:1963 “Sketchpad” I.E Sutherland, "Sketchpad: A Man-Machine Graphical Communication System," Ph.D.

Thesis, MIT, 1962.

1965Digital line drawing

J.E. Bresenham, “Algorithm for computer control of a digital plotter,” IBM Systems Journal, 4(1), 1965.

1971 Gouraud shading H. Gouraud, “Continuous shading of curved surfaces,” IEEE Transactions on Computers, C-20, 1971.

1972Hidden surface removal

M.E. Newell, R.G. Newell, T.L. Sancha, “A Solution to the Hidden Surface Problem,” Proceedings of ACM National Meeting, 1972.

1974 Polygon clipping I.E. Sutherland, G.W. Hodgman, “Reentrant Polygon Clipping,” CACM, 17(1), 1974.

1975 Phong shading B.T. Phong, “Illumination for Computer Generated Pictures,” CACM, 18(6), 1975.

1976Scene graphsTexture mapping

J.H. Clark, “Hierarchical Geometric Models for Visible Surface Algorithms,” CACM, 19(10), 1976.J.F. Blinn, M.E. Newell, “Texture and Reflection in Computer Generated Images,” CACM, 19(10), 1976.

1977ShadowsStandard lighting model

F.C. Crow, “Shadow Algorithms for Computer Graphics,” Computer Graphics, 11(2), 1977.J.F. Blinn, “Models of Light Reflection for Computer Synthesized Pictures,” Computer Graphics, 11(2), 1977.

1978 Bump mapping J.F. Blinn, “Simulation of Wrinkled Surfaces,” Computer Graphics, 12(3), 1978.

HistoryRenderingAuthoring

Page 7: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of surface graphics

• Selected Technology events:1980 Ray tracing T. Whitted, “An improved illumination model for shaded display,” CACM, 23(6), 1980.

1982Geometry VLSIAutoCAD

J.H. Clark, “The Geometry Engine: A VLSI geometry system for graphics,” SIGGRAPH 82.Autodesk Inc.

1983Particle systemsAlias

W.T. Reeves, “Particle Systems – A Technique for Modeling a Class of Fuzzy Objects,” SIGGRAPH 83.Alias Research Inc.

1984

RadiosityShadersSGI IRIS 1000Wavefront

C.M. Goral, K.E. Torrance, D.P. Greenberg, B. Battaile, “Modeling the Interaction of Light Between Diffuse Surfaces,” SIGGRAPH 84.R.L. Cook, “Shade trees,” SIGGRAPH 84.Silicon GraphicsWavefront Inc.

1986 Rendering eq. J.T. Kajiya, “The rendering equation,” SIGGRAPH 86.

1989 RenderMan PIXAR

1992 OpenGL OpenGL ARB

1993 Direct3D Microsoft

1994 GLINT 300SX 3Dlabs

1995 VRML VRML/Web3D Consortium

1997VoodooRiva 128

3dfxnVidia

HistoryRenderingAuthoring

Too many papers to list

Page 8: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of surface graphics

• Selected Story Telling events:1982

TronWrath of Khan

DisneyParamount

1984Andre & Wally B.The Last Starfighter

PIXARUniversal

1985 Young Sherlock Holmes Paramount

1986 Luxo Jr. PIXAR

1987Red’s DreamStanley & Stella

PIXARSymbolics

1988 Tin Toy PIXAR

1989KnickknackThe Abyss

PIXARFox

1991Beauty and the BeastTerminator 2

DisneyTri-Star

1992AladdinLawnmower Man

DisneyNew Line

HistoryRenderingAuthoring

All images copyright by Disney, Fox, Paramount, New Line, PIXAR, and Tri-Star, as appropriate.

Page 9: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of surface graphics

• Selected Story Telling events:1993 Jurassic Park Universal

1994The MaskRebootTrue Lies

New LineMainFrameFox

1995 Toy Story PIXAR

1996DragonheartTwister

UniversalWarner Bros.

1997

Geri’s GameLost WorldStar Wars, Special EditionTitanic

PIXARUniversalLucasfilmParamount

1998A Bug’s LifeAntz

PIXARDreamWorks

1999Toy Story 2Star Wars, Episode 1

PIXARLucasfilm

2000 Dinosaur Disney

HistoryRenderingAuthoring

All images copyright by Disney, DreamWorks, Paramount, Lucasfilm, MainFrame, PIXAR, and Universal, as appropriate.

Page 10: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of surface graphics

• Selected Game events:1991 Catacomb 3-D Id Software

1992Wolfenstein 3-D7th Guest

Id SoftwareTilobyte Studios

1993DOOMMyst

Id SoftwareCyan Productions

1995 Dark Forces LucasArts

1996Nintendo 64QuakeTomb Raider

NintendoId SoftwareEidos Interactive

1997Jedi KnightQuake IIRiven

LucasArtsId SoftwareCyan Productions

1998Half-LifeThiefUnreal

SierraEidos InteractiveEpic MegaGames

1999 Quake III: Arena Id Software

2000 RealMyst Cyan Productions

Catacomb 3-D

DOOM

Quake

Unreal

Jedi Knight

HistoryRenderingAuthoring

Tomb Raider

Half-Life

Quake III: Arena

All images copyright by Id Software, Cyan Productions, Eidos Interactive, Epic MegaGames, LucasArts, and Sierra, as appropriate.

Page 11: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of surface graphics

1960’s 1970’s 1980’s 1990’s 2000’s

TechnologyTechnology

Story TellingStory Telling

GamesGames

HistoryRenderingAuthoring

Page 12: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of volume graphics

• Selected Technology events:1984

Volume ray casting

H. Tuy, L. Tuy, “Direct 2D display of 3D objects,” Computer Graphics and Applications, 4(10), 1984.

19853D texture generation

K. Perlin, “An image synthesizer,” SIGGRAPH 85.

1987 Marching cubes W.E. Lorensen, H.E. Cline, “Marching cubes: A high resolution 3D surface construction algorithm,” SIGGRAPH 87.

1989 Hypertexture K. Perlin, E.M. Hoffert, “Hypertexture,” SIGGRAPH 89.

1990 Splatting L. Westover, “Footprint evaluation for volume rendering,” SIGGRAPH 90.

1991 Volume sculpting T.A. Galyean, J.F. Hughes, “Sculpting: An interactive volumetric modeling technique,” SIGGRAPH 91.

1994Shear-warp3D texture mapping

P. Lacroute, M. Levoy, “Fast volume rendering using a shear-warp factorization of the viewing transformation,” SIGGRAPH 94.B. Cabral, N. Cam, J. Foran, “Accelerated volume rendering and tomographic reconstruction using texture mapping hardware,” Volume Visualization Symposium, 1994.

1999VolumeProGeForce 256

Real-Time Visualization, MitsubishinVidia

2000GeForce2DirectX 8.0

nVidiaMicrosoft

HistoryRenderingAuthoring

Too many papers to list

Page 13: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of volume graphics

• Selected Story Telling events:1997 Contact Warner Bros.

1998 Sphere Warner Bros.

HistoryRenderingAuthoring

Contact

Sphere

All images copyright by Warner Bros., San Diego Supercomputer Center, and the American Museum of Natural History, as appropriate.

Page 14: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of volume graphics

• Selected Game events:2001 ?

?

HistoryRenderingAuthoring

Page 15: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of volume graphics

1960’s 1970’s 1980’s 1990’s 2000’s

TechnologyTechnology

Story TellingStory Telling

GamesGames

HistoryRenderingAuthoring

Page 16: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Evolution of volume graphics

• Volume graphics today……is where surface graphics was 15 years ago

– We are at the start of a transition from technology to tool

• What enabled story telling and games for surface graphics?

• What might do the same for volume graphics?

HistoryRenderingAuthoring

Page 17: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

• 1st business-level (expensive):– Attention-grabbing event: TRON

– Interactive rendering hardware: SGI

– Authoring tools: Alias, Wavefront

1982-4 1991-3 1997

• 1st consumer-level (cheap):– Attention-grabbing event: Catacomb3D, DOOM– Interactive rendering hardware: Voodoo

Enabling eventsHistoryRenderingAuthoring

Surface graphics1960’s 1970’s 1980’s 1990’s 2000’s

TechnologyTechnology

Story TellingStory Telling

GamesGames

Page 18: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

• 1st business-level (expensive):– Attention-grabbing event: ?

– Interactive rendering hardware: VolumePro, SGI?

– Authoring tools: ?

1999

?2000

• 1st consumer-level (cheap):– Attention-grabbing event: ?– Interactive rendering hardware: nVidia?

Enabling eventsVolume graphics

HistoryRenderingAuthoring

1960’s 1970’s 1980’s 1990’s 2000’s

TechnologyTechnology

Story TellingStory Telling

GamesGames

Page 19: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

HistoryRenderingAuthoring

Interactive Volume Rendering…

HistoryRenderingAuthoring

• What can we do now?

• What would we like to do?

• What might we be able to do in the future?

• How might this affect the way we render volumes?

Page 20: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What can we do now?

• Canonical volume visualizations…

• What can we do interactively?

HistoryRenderingAuthoring

Page 21: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What can we do now?

RTviz VolumePro

nVidia GeForce2Ultra

SGI IR2(1 pipe, 4 RMs)

Memory 256 MB 64 MB 64 MB

Fill rate N/A 1 Gpixels/s 896 Mpixels/s

Texture rate N/A 2 Gtexels/s 768 Mtexels/s

Max stored volume

16 x 2563

(scalar)2563

(RGBα)2563

(RGBα)

Max volume 2563 @ 30 fps 2563 @ 15 fps 2563 @ 12 fps

• But 2563 is a small volume…

HistoryRenderingAuthoring

Data source: Product literature

Page 22: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What resolution can we see?

• Eye’s lens focuses light onto retina– Fovea = focus area = center 20

• More receptors at the fovea– Mostly “Cones” (color) at fovea

– Mostly “Rods” (intensity) in surrounding areaFovea

Periphery

HistoryRenderingAuthoring

Data source: Perception, Sekuler & Blake, 2nd ed, 1990, McGraw-Hill.

Page 23: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What resolution can we see?

• Measure visual acuity by viewing “gratings” of parallel lines• How fine a grating can you see?

HistoryRenderingAuthoring

Gratings

Data source: Visual Perception, Spillmann & Werner, 1990, Academic Press.

• Normal vision:• 120 lines/degree in center 20 = 2,400 lines

• Legally blind:• 1/10th normal vision

• 12 lines/degree in center 20 = 240 lines

Page 24: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What resolution can we see?

• A 2563 volume legally blind!– Very low resolution

• Normal vision requires 2,4003!– If it only covers central 20 of visual field

HistoryRenderingAuthoring

• 180 visual field requires 21,6003!– 120 lines/degree in 180 = 21,600 lines!

Page 25: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What resolution can we show?

• We’re constrained by screen resolution– 1280x1024 on a large monitor

– 30 pixels/degree at normal distance

– ¼ normal vision = visually impaired

• For now…– 10243 is a minimum for effective screen use

• Smaller volumes are blurry (1 voxel covers multiple pixels)

– But we want much higher volume resolutions…

HistoryRenderingAuthoring

Page 26: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What resolution do we want?

• Medical imaging (cryosections)– State of the art: 2048 x 2048 images, 1/10mm apart

– 2048 x 2048 x 18000 for full body (180cm person)

– Visible Human Male: 2048 x 1216 x 1871• Re-digitization of images: 4096 x 2700 x 1871

– Visible Human Female: 2048 x 1216 x 5189

– Recent brain only: 1789 x 1472 x 1152

HistoryRenderingAuthoring

Head data from the Visible Human Project, www.nlm.nih.gov/research/visibleBrain data from Arthur Toga’s lab, LONI, UCLA, www.loni.ucla.edu

Page 27: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What resolution do we want?

• Medical imaging (CT scan)– State of the art: 2048 x 2048 images, ½mm apart

– 2048 x 2048 x 3600 for full body (180cm person)

– Visible Human Male: 512 x 512 x 1871

HistoryRenderingAuthoring

Head data from the Visible Human Project, www.nlm.nih.gov/research/visible

Page 28: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What resolution do we want?

• Microscopy imaging (fluorescence)– State of the art: 1024 x 1024, 0.15 microns apart

– 1024 x 1024 x 75 for a 10 micron cell

– Cell membranes: 768 x 768 x 72

HistoryRenderingAuthoring

Cell data from Hiro Tsukada, Eric Elenko, and Maria Pinhal, UCSD Cancer Center

Page 29: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What resolution do we want?

• Simulations– Tend to use all available memory on a supercomputer

– SDSC IBM SP “Blue Horizon”• 1152 processors (144 nodes, 8 cpus/node, +others), 1.7 teraflops

• 576 Gbytes total memory (4 Gbytes/node)

• Largest possible volume is 50003

– Ocean simulation: 2160 x 960 x 30

HistoryRenderingAuthoring

Ocean data from Detlef Stammer, UCSD, Scripps Institute of Oceanography

Page 30: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What resolution do we want?

• Combine to build volumetric scenes– Composite overlapping volumes

– Mosaic volumes to fill space

– Combine procedural elements

HistoryRenderingAuthoring

Page 31: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What resolution do we want?

110

1001,000

10,000100,000

1,000,00010,000,000

100,000,0001,000,000,000

10,000,000,000100,000,000,000

1,000,000,000,000

Nu

mb

er

of

vo

xe

ls

HistoryRenderingAuthoring

2563

• We’re a long way from what we want… why?

Full screenFoveaFull eye

It’s a logarithmic scale!

Page 32: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Memory and bandwidth

• Memory needed grows with cube of volume’s width– 643 RGBα = 1 Mbyte

– 1283 RGBα = 8 Mbytes

– 2563 RGBα = 64 Mbytes

– 5123 RGBα = 512 Mbytes

– 10243 RGBα = 4096 Mbytes

• Bandwidth needed also grows with frame rate– 10 fps minimum

– 30 fps better

– 70 fps best

HistoryRenderingAuthoring

0

1024

2048

3072

4096

0 256 512 768 1024

Resolution

Mb

yte

s

Page 33: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What will volumes require?

Memory 10 fps 30 fps 70 fps

2563

(legally blind)64 MB 640 MB/s 1.9 GB/s 4.5 GB/s

10243

(full screen)4 GB 40 GB/s 120 GB/s

(30 Gpixl/s)180 GB/s

24003

(fovea)55 GB 550 GB/s 1.6 TB/s 3.8 TB/s

216003

(full eye)40 TB 400 TB/s 120 TB/s 280 TB/s

• 120 GB/s is > 30 times current bandwidths!

HistoryRenderingAuthoring

Page 34: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Growth of pixel fill ratesHistoryRenderingAuthoring

Data source: Product literature

0

200

400

600

800

1000

1200

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

Fill

ra

te, M

pix

els

/s

SGI PC cards

1996 30 Mpixels/s 3Dlabs Permedia1997 100 Mpixels/s nVidia RIVA 1281998 366 Mpixels/s 3dfx Voodoo 31999 540 Mpixels/s ATI Rage Fury MAXX2000 1000 Mpixels/s nVidia GeForce2 Ultra

* Not counting custom hardware or special configurations

• Fill rates recently growing by x2 every year

Page 35: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

How long to get there?

Bandwidth @ 30 fps Years

2563

(legally blind)1.9 GB/s(0.5 Gpixels/s)

Last year

10243

(full screen)120 GB/s(30 Gpixels/s)

5 years

24003

(fovea)1600 GB/s(400 Gpixels/s)

8 years

216003

(full eye)120000 GB/s(30,000 Gpixels/s)

14 years

HistoryRenderingAuthoring

• But, this is not very realistic…

Page 36: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

How long to get there?

About trends…

“A frequent criticism of predictions of the future is that they rely on mindless extrapolation of current trends without consideration of forces that may terminate or alter that trend.”

– Ray Kurzweil, The Age of Spiritual Machines

HistoryRenderingAuthoring

Page 37: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

How long to get there?

About the computer industry…

“If the automobile industry had made as much progress in the past fifty years, a car today would cost a hundredth of a cent and go faster than the speed of light.”

– Ray Kurzweil, The Age of Spiritual Machines

HistoryRenderingAuthoring

Page 38: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

CPU performance growthHistoryRenderingAuthoring

Data source: SPEC benchmark database, www.spec.org

0

10

20

30

40

50

60

70

80

90

1994 1995 1996 1997 1998 1999 2000 2001

SP

EC

fp9

5

• Floating point power grows by x2 every 2 years

Page 39: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

CPU performance growth

• At this rate, by 2020 transistor insulators will be just a few atoms thick– A possible end to growth using this technology

– What technology will arrive next?

• Meanwhile, back to the present…

HistoryRenderingAuthoring

Page 40: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Memory speed growthHistoryRenderingAuthoring

• Memory bandwidth only grows by x2 every 5 years

Data source: Kingston Technology, www.kingston.com

0

20

40

60

80

100

120

140

19

85

19

86

19

87

19

88

19

89

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

Me

mo

ry b

us

sp

ee

d, M

Hz

• PC processors only

Page 41: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Comparing growth ratesHistoryRenderingAuthoring

0

5

10

15

20

25

30

35

40

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

Inc

rea

se

fa

cto

r

Processor performance growth

Memory bus speed growth

Pixel fill rate growth

• Highly unlikely that fill rates can continue to grow faster than processor speed and memory bandwidth

Page 42: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What should we do?

• Don’t bet on rapid fill rate growth to satisfy our volume rendering needs– If fill rate growth drops to track memory bandwidth

growth, full screen volume rendering may arrive in25 years, not 5 years

• And full eye in 70 years!

HistoryRenderingAuthoring

• Faster hardware is no substitute for a better algorithm– How can we change the way we work?

– Lots of ways… but I’ll focus on a few…

Page 43: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Change how we render

• Object-space rendering traversals are in common use– Scan through all voxels and project towards screen

• Splatting, shear-warp, 3D texture mapping, point clouds

– Memory streaming is possible• If the viewpoint is outside the volume

• O(n3) time & spacen = volume width

HistoryRenderingAuthoring

Page 44: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Change how we render

• Image-space traversals are more time/space friendly– Scan through all screen pixels and project towards voxels

• Ray casting, …

– Memory streaming not possible• Nearly random access

• O(k r n) time & spacen = volume width

r = image resolution (width x height)

k = additional computation factor (for comparisons)

HistoryRenderingAuthoring

Page 45: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

• Is volume ray casting really O(k r n)?– Worst case: each ray takes longest path = n

– Normal case: early ray termination reduces it

– Rays diverge, skipping voxels, introducing aliasing

– For accuracy: supersample

– For speed: use volume MIP-mapping• Upfront cost to build MIP-map volumes

• Increases memory needed (but not bandwidth)– Memory is cheap, bandwidth is not

Change how we renderHistoryRenderingAuthoring

3

Page 46: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

• To interactively render larger volumes, we need to get on the better curves – such as ray casting

Rendering time growthHistoryRenderingAuthoring

0

10

24

20

48

30

72

40

96

51

20

61

44

71

68

81

92

Volume size

Re

nd

eri

ng

Tim

e

Object space traversalImage space traversal (k=5)Image space traversal (k=10)Image space traversal (k=15)Image space traversal (k=20)

full

scr

een

fove

a

lega

lly

bli

nd

Page 47: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

What can we do?

• Changing our rendering algorithm can help– For “large data,” image-space traversals are better than

object-space traversals

– We have “large data” now

HistoryRenderingAuthoring

110

1001,000

10,000100,000

1,000,00010,000,000

100,000,0001,000,000,000

10,000,000,000100,000,000,000

1,000,000,000,000

VH Mal

e Cry

osect

ion

VH Mal

e (n

ew) C

ryose

ctio

n

VH Fem

ale

Cryose

ctio

n

Brain

Cry

osect

ion

Full Body

Cryose

ctio

n

VH Mal

e CT

Full Body

CT

Mem

brane

Mic

rosc

opy

Full Cel

l Mic

rosc

opy

Ocean

Sim

ulatio

n

SDSC Max

Sim

ulatio

n

Nu

mb

er

of

vo

xe

ls

Cross-over

• What about changing our data?

Page 48: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

HistoryRenderingAuthoring

Volume Authoring…

HistoryRenderingAuthoring

• How might rendering issues affect how we create volumes?

• What might future authoring tools look like?

Page 49: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

CPU vs. memory speedsHistoryRenderingAuthoring

• PC processors, excluding RDRAM & DDR SDRAM

• Main memory access (cache miss)

• Slower memory speed growth means memory references are getting more costly, in terms of cycles

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

1994 1995 1996 1997 1998 1999 2000 2001

CP

U M

Hz

/ Me

mo

ry M

Hz

Data source: SPEC benchmark database, www.spec.org

Page 50: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

CPUs per SupercomputerHistoryRenderingAuthoring

99 117 128173

226 281230 281424

581 687 806

1

10

100

1000

10000

1994 1995 1996 1997 1998 1999 2000 2001

Nu

mb

er

of

pro

ce

ss

ors

Average of top 100

Average of top 500

• Adapt to slow memory by adding CPUs w/own memory• CPUs/Supercomputer grows by x2 every 3 years

Data source: Top 500 Supercomputers, www.top500.org

Page 51: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Cycles & bandwidth

• Cycles are more available than memory bandwidth– “Easier” to add CPUs than increase bandwidth

– Parallel computing is an obvious industry trend• But CPU coordination and data sharing is still bandwidth limited

HistoryRenderingAuthoring

• Can we trade computation for memory accesses?– Data compression is clearly needed

• Texture compression nearly standard in graphics hardware

• Geometry compression becoming available

– Additional techniques to reduce memory access costs• Interleaved frame buffers, Z-buffer cache

• View frustum culling, occlusion culling

Page 52: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Authoring with cycles

• “Shaders” compute parts of the scene as needed– Already common-place in software rendering

• “Data amplification” – small number of parameters generates large amount of scene content

– Programmable graphics pipelines arriving now

– For surface graphics: splines, procedural textures, particles

– For volume graphics: voxelization, procedural volumes

HistoryRenderingAuthoring

Page 53: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Authoring with cycles

• Shaders aren’t just for shading any more– Procedural content creation

• Noise and turbulence functions

– Voxelization based upon geometry “parameters”• Voxelize on demand – don’t prevoxelize

• Authoring uses a mix of data and shaders– Import data sets (CT, MRI, cryosection, etc.)

– Sculpt & 3D paint volumes

– Program “shaders”

HistoryRenderingAuthoring

Page 54: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Authoring with shaders

• Procedural turbulence creates “clouds”– Defines a color & opacity at each point in space

– Animate parameters to evolve the cloud

– Voxelize during ray-casting• No volumes created

• Very low memory use

HistoryRenderingAuthoring

Page 55: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Authoring with shaders

• Manipulate shapes– Compute effects during ray-casting

• No additional volumes created

• For each point in space:– Compute shortest distance to surface

– Perturb the distance with turbulence

– Map distances to opacity

HistoryRenderingAuthoring

Compute distances Add turbulence Map to opacity Do at high resolution

Page 56: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Authoring with shaders

• Constructing shapes– Constructive Solid

Geometry (CSG)

– Cut using primitive shapes• (or anything)

– Transform and “cut”during ray-casting

• No pre-cut volumes created

HistoryRenderingAuthoring

Page 57: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Authoring with shaders

• Repeat to build volumetric scenes– Multiple data sets, geometry, shaders, …

– Evaluate some or all during ray-casting• Rendering is an active part of authoring

HistoryRenderingAuthoring

Page 58: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Authoring scenes

• Several existing scene structure metaphors– CSG trees = combine primitives (most CAD apps.)

– Composite trees = combine images (Houdini, Shake)

– Shade trees = combine shaders (RenderMan)

– Scene graphs = lay out data sets (VRML, Java3D)

– Expression trees = compute scene (any programming lang.)

• “Compilation” prevoxelizes some, all, or none of the scene– Tune to minimize error, memory use, rendering time

HistoryRenderingAuthoring

Page 59: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Visualizing the Orion nebula

• Fluorescing clouds of hydrogen, helium, oxygen, ...– Complex structure with pillars, swirls, ripples, and cavities

Eagle Nebula Lagoon Nebula Orion Nebula

HistoryRenderingAuthoring

Page 60: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

• Build a surface for the ionization front– Derived from visual and infrared data

• Make it fuzzy– Perturb distance field with turbulence

• Project a color-corrected Hubble image through it– Jitter to reduce streaking

Visualizing the Orion nebulaHistoryRenderingAuthoring

Page 61: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Visualizing the Orion nebula

• Add 85 proplyds (protostars), shock fronts, …– Each built in a similar way

• Fly around in it all

HistoryRenderingAuthoring

Page 62: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Visualizing the Orion nebulaHistoryRenderingAuthoring

Page 63: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Visualizing the Orion nebula

• 2112 shader nodes in the full scene– Surfaces, turbulence, image projection, transforms, …

• Prevoxelized as 86 volumes– 2 Gbytes of multi-resolution data

– 40 Gbytes if we didn’tvoxelized separately

• Ray-caster composited volumes and stars

HistoryRenderingAuthoring

Page 64: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Concluding remarks

• What are some directions to explore?

HistoryRenderingAuthoring

Page 65: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Data directions

• Volume registration– Align volumes for compositing

• Volume compression– Make it take less space

• Volume decimation– Express it well at a smaller size

“Multimodal Medical Volume Registration Based on Spherical Markers”

“Geometric Processing of Volumetric Objects”

“Exploiting Eigenvalues of the Hessian Matrix for Volume Decimation”

At WSCG 2001

“Towards Continuous Image Representations”

Page 66: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Rendering directions

• Rendering from compressed data– Smaller data = less bandwidth needed

• Large (out-of-core) data rendering– I/O during rendering to get data

• Parallel rendering– More processors to get more bandwidth

• Hybrid volume + geometry rendering– Keep shapes in their smallest format

“A New Parallel Volume Rendering Algorithm”

At WSCG 2001

“Parallel Ray Tracing with 5D Adaptive Subdivision”

Page 67: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Authoring directions

• Content creation “shaders”– Create using procedures

• Volume sculpting / 3D painting– Create from scratch in an artist-natural way

• Volume scene construction– Compose, composite, cut, …

• Volume scene compilation– Prevoxelize for best memory/CPU use

At WSCG 2001

“Hypertexturing Complex Volume Objects”

Page 68: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Tool directions

• Work with users!– Put technology into user’s hands … make it a tool

– Help create that attention-grabbing event

?

Page 69: SAN DIEGO SUPERCOMPUTER CENTER University of California, San Diego Volume Graphics Technology to Tools David R. Nadeau Principal Scientist San Diego Supercomputer

SAN DIEGO SUPERCOMPUTER CENTERUniversity of California, San Diego

Acknowledgements

• USA National Science Foundation

• USA Department of Energy

• San Diego Supercomputer Center– Mike Bailey

– Steve Cutchin

– Alex DeCastro

– Eric Enquist

– Jon Genetti

– Mike Houston

– John Moreland