國立臺灣大學電機資訊學院資訊工程學研究所

碩士論文Department of Computer Science and Information Engineering

College of Electrical Engineering and Computer Science

National Taiwan UniversityMaster Thesis

支援似鏡面材質的直接—間接轉移打光系統Glossy Direct-to-Indirect Transfer for Relighting

楊善詠Yang, Shan-Yung

指導教授：莊永裕博士Advisor: Chuang, Yung-Yu, Ph.D.

中華民國 96年 7月July, 2007

國立臺灣大學

資訊工程學研究所

碩士論文

支援似鏡面材質的直接—間接轉移打光系統

楊善詠撰

967

致謝

常聽到「感謝父母為我生了一對好手好腳」的說法，卻從來沒聽過有人說「感謝父母給我一顆健全的大腦」。對於能夠順利完成論文的自己，我十分感謝雙親所贈與我的一切，包括健全的家庭教育、無盡的包容與支持、還有這顆雖然不怎麼聰明，卻能運作無誤的大腦。在進行研究的這段期間，協助我最多的自然是指導教授莊永裕老師。老師給我

的不只是豐富的知識，也包括嚴謹而科學的態度和獨樹一格的幽默，至今我還記得老師說過的一句話：「小善你還蠻聰明的嘛，出乎我的意料之外。」希望這篇論文與致謝詞也能出乎他的意料。而我最珍貴的朋友們：葉王、小金、昊極、昌熹、子桓 (依照 ID排序)，能和

你們一起做研究、一起吃飯、一起幫某人慶生，我感到三生有幸。也感謝京都動畫公司製作的「らきすた」，在最後這幾個月提供我大量的吐槽用例句，諸如「嗚嗚，可以吐槽的地方太多了」或是「我現在很懶，可以不吐你槽嗎」之類。在大家一起趕論文時，這些句子實在相當好用。正經八百的致謝文到此為止，因為這一頁還有很多空間，就讓我好好利用一下

這塊空間吧。首先感謝先天不良、後天失調的 C++，我總是在一些枝微末節的地方花費最多

時間。而 PBRT更是讓這個問題雪上加霜，BSDF這個 class在記憶體管理上毫無彈性可言，一不小心就會 double free、memory leak或是誤用已被釋放的記憶體。雖然「理論上」我只需要把我的演算法寫成 plug-in，但實際上我還是被迫去 hack一堆核心部份的程式碼，如果有人想搞懂這部份，我只能說他勇氣可嘉。接著感謝 NVIDIA R© 問題多多的驅動程式。在 Linux上使用 7800在切換 FBO

時效率極差，即使用了 100.14.11 版的驅動程式還是沒解決這個問題。而且在使用 glBlendFunc(GL ONE, GL ONE) 累加結果時，如果輸出是一張 floatingpoint texture，加起來的結果根本是錯的。同樣地，要是有學弟妹們想玩 GPUprogramming，我也會拍拍他的肩膀，稱讚他勇氣可嘉。此外，感謝台大圖書館莫名其妙的論文政策，為什麼我們上傳論文前還要把

PDF 中的文字複製功能鎖住？只要 PDF 出現在螢幕上，不用複製貼上同樣也可以把內容抄下來，關閉複製功能只是防止偷懶的大一生抄論文當報告罷了，對散佈與保存知識一點幫助也沒有。對此我決定拒絕授權圖書館的電子全文服務，除非他們開放文字複製功能。還有印精裝本的規定實在是浪費地球資源，為了印精裝本的側邊，使得有些頁數不夠的論文 (好啦，就是我)必須印單面，對於這個問題，我還一度想在附錄中加入立體紙雕好增加厚度。我實在很想對做出這決定的人提出建議：「地球的空氣已經很髒了，能不能請你們不要呼吸？」抱怨了這麼多，其實這一頁真的只是致謝文，只不過它是唯一可以讓我自由發

揮的地方。因為篇幅規定只有一頁，紙短情長，我還是只能就此打住。最後我要感謝的是閱讀這篇論文的你，我沒想過這篇論文送進圖書館後會有人翻開它。希望論文內容可以給你一點幫助，如果想吐我的槽，也儘管放馬過來。

楊善詠 7-17-2007於台大資訊系 R503

i

ii

摘要

在包含數位特效的電影或電腦動畫中，燈光設計一直是個耗時又耗力的過程。燈光設計師需要一套互動式的工具讓他們得以即時預覽結果，但又不希望預覽與最終繪製的結果差異過大。在這篇論文中，我們提供了一套解決燈光師難題的打光系統。藉由

事先運算並儲存的著色快取影像，再加上繪圖硬體的加速，這套系統可以快速產生品質良好的打光預覽圖。我們也使用多層著色快取影像的方式，提供了鏡面反射的效果。全域照明也是這篇論文的重點。我們提出了一套根基於光能採集點

的演算法，並藉由採集點間的能量交換來計算間接照明能量。我們的主要貢獻在於對雙向反射分佈函數進行分解，比起傳統假設材質皆為漫射面的方法，我們的演算法能更有效地模擬似鏡面材質間的能量傳遞。

iii

iv

Abstract

In computer cinematography and animation, the process of lighting design isboth time consuming and labor intensive. Lighting artists need an interactivetool which gives fast preview without losing too much quality compared withthe final result.

In this thesis, we present a relighting system which solves the problemfaced by lighting artists. By caching shading parameters in deep framebufferand rendering with graphics hardware acceleration, our system gives con-vincible lighting preview interactively. Our system also captures specularreflections by layered deep framebuffer.

Global illumination for relighting system is also emphasized in this thesis.We propose a new approach based on transferring indirect illumination fromgather samples. Our main contribution is extending the light transport path toglossy inter-reflections by BRDF factorization.

v

vi

Contents

致謝 i

摘要 iii

Abstract v

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Related Work 32.1 Relighting engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Precomputed radiance transfer . . . . . . . . . . . . . . . . . . . . . . . 32.3 Global illumination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Relighting System 53.1 Deep framebuffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53.2 Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53.3 Light scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.4 Specular reflections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.5 Shadows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4 Global Illumination 114.1 Basic derivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.2 Diffuse direct-to-indirect transfer . . . . . . . . . . . . . . . . . . . . . . 134.3 Glossy direct-to-indirect transfer . . . . . . . . . . . . . . . . . . . . . . 154.4 BRDF factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5 Experiments and Results 215.1 Relignting system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215.2 Glossy direct-to-indirect transfer . . . . . . . . . . . . . . . . . . . . . . 24

6 Conclusion and Future Work 29

Bibliography 31

vii

viii

List of Figures

3.1 Data structure of deep framebuffer. . . . . . . . . . . . . . . . . . . . . . 6

3.2 Relighting process on GPU. . . . . . . . . . . . . . . . . . . . . . . . . 7

3.3 Block diagram of our relighting system. . . . . . . . . . . . . . . . . . . 10

4.1 Relationship between p and p′ in Equation 4.2. . . . . . . . . . . . . . . . 12

4.2 Projection from hemispherical directions to texture coordinates. . . . . . 18

5.1 Test scene with one diffuse sphere and four specular ones. . . . . . . . . 22

5.2 Test scene with complex models. . . . . . . . . . . . . . . . . . . . . . . 23

5.3 Cornell box by different illumination algorithms. . . . . . . . . . . . . . 25

5.4 Indirect illumination by glossy-only BRDFs, scaled by 4. . . . . . . . . . 27

5.5 Buddha by different illumination algorithms. . . . . . . . . . . . . . . . . 28

ix

x

List of Tables

3.1 Pseudo codes of the generation of deep framebuffer. . . . . . . . . . . . . 6

3.2 Modified algorithm to generate layered deep framebuffer. . . . . . . . . . 9

5.1 Rendering time of our relighting system and Blender. . . . . . . . . . . . 21

5.2 Rendering times of difference global illumination algorithms. . . . . . . . 26

xi

xii

Chapter 1

Introduction

1.1 Motivation

Lighting design is a time consuming and labor intensive process in modern production

rendering. Lighting artists need fast and accurate feedback when they modify light pa-

rameters, but often forced to accept either long render time or poor image quality. The

major challenge in providing interactive feedback is the scene complexity in modern pro-

duction, which results in huge computation.

Many relighting systems have been presented to speed up scene relighting while main-

taining quality comparable to that of the production renderer. These relighting systems

usually focus on two approaches of acceleration. One is complexity reduction by as-

suming fixed views and static scenes, allowing shading parameters such as positions

and normals cached in image space. The other approach is to exploit graphics process-

ing units (GPUs) which are evolving rapidly and becoming an attractive platform for

computation-intensive tasks. When the user modifies light parameters, these relighting

systems are capable to updates the result interactively by hardware accelerated shaders.

Current state-of-the-art relighting system, direct-to-indirect transfer (DTI) [8], sup-

ports indirect illumination and still provides interactive frame rates in complex scenes.

However, the light transport path in DTI is limited to diffuse inter-reflections. The in-

direct illumination produced by DTI for scenes containing glossy objects is not accurate

when compared with the final result.

1.2 Problem statement

Given a scene data which may contain high geometric complexity, lighting artists need a

fast preview when configuring light parameters. The system can be described by follow-

ing characteristics:

1

• Interactivity. The primary goal of our system is to provide lighting artists interac-

tive feedback, which is the key of productivity.

• Accuracy. In order to make the result meaningful to artists, correct light falloff

and surface response are necessary. Accurate positioning of shadows and specular

reflection is also critical in the relighting process.

• Scalability. In production rendering there are usually at least 10–20 light sources.

Our system must maintain scalable performance while the number of light sources

increases.

1.3 Organization

In chapter 2, previous work on relighting systems is reviewed. The design of our relight-

ing system satisfying the problem statement mentioned above is described in chapter 3.

We also propose an extension to DTI which supports glossy inter-reflections in chapter 4.

The experiments and results are discussed in chapter 5. The last part, chapter 6, contains

conclusion and future direction of our work.

2

Chapter 2

Related Work

Their are two major parts in this thesis: one is an interactive relighting engine imple-

mentation which supports specular reflections, and the other is a new approach for global

illumination in relighting systems. The previous work can be roughly classified into three

categories: relighting engines, precomputed radiance transfer and global illumination.

2.1 Relighting engines

Most relighting engines are based on the idea of deep framebuffers. This idea can be found

in G-buffer [19] which is a collection of images containing geometric data for rendering.

Sequin and Smyrl [20] stored the data in tree structures corresponding to their usage in

ray tracing. This is extended in ray trees [4] to allow small visibility changes.

Advances of graphics hardware make it possible to evaluate the result interactively.

Gershbein and Hanrahan [7] exploited the fixed function unit of graphics hardware for

evaluation and still supports some common bidirectional reflectance distribution func-

tions (BRDFs). The shading flexibility is extended in Lpics [15] by programmable graph-

ics hardware. Due to the limitation of GPUs, Lpics only supports direct illumination

model. Hasan et al. [8] made use of gather samples and achieved real-time indirect illu-

mination. Ragan-Kelley et al. [17] further automated the translation of hardware shaders

by data-flow analysis.

Although the indirect illumination can be achieved by so-called “direct-to-indirect

transfer” on gather samples, the light transport path is limited to diffuse inter-reflections. [8]

2.2 Precomputed radiance transfer

Recently, there has been much work aimed at relighting static scenes under distant illumi-

nation. This was pioneered by Sloan et al. [21] where low-frequency distant illumination

3

is represented by spherical harmonics. Ng et al. [13] extended this work to all-frequency

illumination by wavelet approximation of environment map. However, this introduced

a problem that wavelet basis is not rotatable like spherical harmonic basis, which disal-

lows glossy inter-reflections. This problem can be solved by triple product [14] or BRDF

factorization. [12][22]

Although some PRT algorithms supports “mid-range” lighting [2], they do not fully

meet cinematic relighting where point lights and spot lights are widely used. Rather than

using PRT directly, we exploit BRDF factorization to extend the light transport path in

direct-to-indirect transfer.

2.3 Global illumination

Most algorithms dealing with global illumination, such as irradiance caching [23] and

photon mapping [9], assume that the distribution of indirect illumination among the scene

is smooth. This allows caching indirect lighting in a sparse set of samples and reconstruct

the illumination during the rendering pass. In these algorithms the cached information

often depends heavily on light sources, and becomes invalid when the light parameters

are modified.

4

Chapter 3

Relighting System

This chapter describes the detail of our relighting system which supports direct illumina-

tion model as well as specular reflections.

3.1 Deep framebuffer

We adopt the assumption of deep framebuffer-based algorithm that both camera and scene

objects are fixed during the relighting process. As pointed out by Pellacini et al. [15],

geometric complexity has grown dramatically in recent years, but image resolution grows

at a much slower pace. By assuming fixed camera and scene it is possible to cache shading

parameters in image space and makes the rendering time proportional to image resolution

instead of geometry complexity.

The image cache, which is called deep framebuffer, has dimensions as same as those

of the final rendered result. The data stored in deep framebuffer are shown in Figure 3.1.

Note that the actual stored data depend on reflection models used in the scene. All data

except light parameters necessary for shading computation must be cached in deep frame-

buffer. By a modified ray casting algorithm, the generation of deep framebuffer is quite

straightforward as described in Table 3.1.

3.2 Rendering

In the relighting process, light parameters are provided by users. For each pixel in deep

framebuffer, the rendering result is computed from cached shading parameters and light

parameters. Since this process is computation-intensive with data parallelism, we can

further improve the rendering speed by exploiting GPUs.

However, GPUs are dedicated to rasterization graphics and the programming model

differs from that on CPUs. The following steps describe how relighting computation is

5

Scene

Camera Image Plane

Stored shading paramters:

Weight W

Position (Px, Py, Pz)Normal (Nx, Ny, Nz)Diffuse Kd

Specular Ks

Roughness r...

...

Figure 3.1: Data structure of deep framebuffer.

GENERATE-DEEP-FRAMEBUFFER(Scene, Camera, Width, Height)

1 initialize D as an image of Width×Height pixels

2 for j ← 1 to Height

3 do4 for i ← 1 to Width

5 do6 r ← GENERATE-EYE-RAY(Camera, i, j)

7 x ← INTERSECT(Scene, r)

8 if x does not exist

9 then10 set the weight of D[i, j] to 0

11 else12 D[i, j] ← shading parameters on x

13 set the weight of D[i, j] to 1

14

15 return D

Table 3.1: Pseudo codes of the generation of deep framebuffer.

6

Vertex Shader Fragment Shader Framebuffer

Quad Map to image size

Render Result

Light Parameters

Position (Px, Py, Pz)Intensity (Ir, Ig, Ib)...

...

Deep Framebuffer

Figure 3.2: Relighting process on GPU.

achieved on GPUs, as illustrated in Figure 3.2.

1. Generate a quad and bind deep framebuffer as its texture.

2. Map this quad to framebuffer so its size matches the dimensions of the rendering

result in vertex shader.

3. In fragment shader, the shading parameters can be obtained by texture fetch from

deep framebuffer. With light parameters the rendering result is computed and stored

in the framebuffer.

3.3 Light scalability

If there are more than one light source, the steps mentioned above must be applied to all

light sources and the final result is the accumulation of the rendered result of each light.

Since the rendering time grows linearly with the number of light sources, it is hard to

achieve interactivity in production relighting where there are at least 10–20 light sources.

Nevertheless, it is natural to suppose that users only modify parameters of one light

source at once. Therefore the rendered results of other light sources can be cached. The

rendering time is reduced to updating one result of the modified light source and accu-

mulation with other cached results. Since accumulation of multiple images on GPUs is

7

much faster than rendering one result from light parameters, this makes our system main-

tain interactive frame rate when the number of light sources increases.

3.4 Specular reflections

Specular reflections are very common in computer animated films. Perfect specular ma-

terials such as mirrors and glossy metals can reflect nearby objects. A Whitted-style ray

tracer [24] can simulate specular reflections by tracing secondary rays, but it requires too

much computation for an interactive application.

Instead of tracing secondary rays, we use layered deep framebuffer for specular reflec-

tion. Since both camera and scene are fixed in our scenario, all secondary rays perfectly

reflected by specular materials are also fixed. Therefore we trace these secondary rays and

store shading parameters in another layer of image cache in the precomputation stage. Ta-

ble 3.2 shows the modified algorithm to generate layered deep framebuffer. `MAX is a user

specified parameter denoting the maximum allowed depth in Whitted-style ray tracing.

In the relighting process, each layer of deep framebuffer is used to render one image as

described in Section 3.2. Each result image is then multiplied by its weight and summed

up. Although the rendering time and required memory for deep framebuffer grow linearly

with `MAX , it is generally sufficient to obtain satisfying results with `MAX = 4.

3.5 Shadows

Shadows are important for reality and emphasized in the relighting process. In our system,

shadow maps [25] are used to decide whether a shading point is lit by light sources.

Although some techniques such as shadow volumes [5] produce more accurate shadows,

there are some advantages of shadow maps:

• It is robust and used widely in 3D games. This makes shadow map generation on

recent GPU very fast.

• Shadow volumes are only available for shading points inside the viewing frustum.

If layered deep framebuffer is applied, shading points on layers other than the first

one may be located outside of the viewing frustum.

The time required for shadow map generation grows linearly with scene complex-

ity. However, it is possible to optimize this step by geometry simplification or occlusion

culling.

As a summarization, the whole relighting system we proposed in this chapter is de-

scribed in Figure 3.3.

8

GENERATE-LAYERED-DEEP-FRAMEBUFFER(Scene, Camera, Width, Height)

1 initialize D as a set of images of Width×Height pixels

2 for j ← 1 to Height

3 do4 for i ← 1 to Width

5 do6 r ← GENERATE-EYE-RAY(Camera, i, j)

7 x ← INTERSECT(Scene, r)

8 w ← 1

9 for ` ← 1 to `MAX

10 do11 if x does not exist

12 then13 set the weight of D`[i, j] . . . D`MAX

[i, j] to 0

14 break15 else16 D`[i, j] ← shading parameters on x

17 set the weight of D`[i, j] to w

18 . Multiply weight by BRDF and cosine term

19 w ← w × f(ωi, ωo) cos θi

20 r ← REFLECT(x, r)

21 x ← INTERSECT(Scene, r)

22

23

24 return D

Table 3.2: Modified algorithm to generate layered deep framebuffer.

9

Mesh Deep Framebuffer

Shadow Map

Light Parameters

Position (Px, Py, Pz)Intensity (Ir, Ig, Ib)...

...

Generateshadow map

Shading

Result

Add with resultsof other lights

Final Result

User

Modify

Preview

Figure 3.3: Block diagram of our relighting system.

10

Chapter 4

Global Illumination

The relighting system described in the previous chapter only supports direct illumination

model. Although specular reflections are provided by layered deep framebuffer, the light

transport path is limited to merely specular inter-reflections.

In this chapter, we focus on global illumination in relighting systems. Our approach

is based on DTI [8] which collects global illumination by gather samples. By applying

BRDF factorization, our approach extends light transport path to both diffuse and glossy

inter-reflections.

4.1 Basic derivation

By the rendering equation [10]:

Lo(p, ωo) = Le(p, ωo) +

∫Ω

f(p, ωo, ωi)Li(p, ωi) cos θidωi (4.1)

where

Lo(p, ωo) is outgoing radiance.

Li(p, ωi) is incoming radiance.

Le(p, ωo) is emitting radiance.

f(p, ωo, ωi) is BRDF on position p.

Ω represents hemisphere.

Let t(p, ω) be the ray tracing function denoting the first surface point intersected by a

ray from p in direction ω. By assuming that there is no participating media, we have:

Li(p, ω) = Lo(t(p, ω),−ω)

We also omit emission term, Le(p, ωo). We do not lose generality by this omission because

emission term is one of material properties and fixed in the relighting process. Therefore

11

p

ωo ωi

n

θi

p′

n′

−ωi

θ′

Figure 4.1: Relationship between p and p′ in Equation 4.2.

we can precompute the result only contributed by emission term and add it to the final

result.

After simplification Equation 4.1 is written as:

Lo(p, ωo) =

∫Ω

f(p, ωo, ωi)Lo(t(p, ωi),−ωi) cos θidωi

By the definition of solid angel ω, we have:

Lo(p, ωo) =

∫Ω

f(p, ωo, ωi)Lo(p′,−ωi) cos θi

cos θ′

‖p − p′‖2dA

=

∫Up

f(p, ωo, ωi)Lo(p′,−ωi)

(ωi · n)(−ωi · n′)

‖p − p′‖2dA (4.2)

where

p′ = t(p, ωi)

n′ is the normal on p′.

Up is the union of surfaces visible from p.

as illustrated in Figure 4.1.

By introducing the geometry term, Equation 4.2 is rewritten as:

Lo(p, ωo) =

∫U

f(p, ωo, ωi)Lo(p′,−ωi)V (p, p′)

(ωi · n)(−ωi · n′)

‖p − p′‖2dA (4.3)

12

where

U is the union of surfaces in scene.

V (p, p′) =

1, if p′ is visible from p.

0, otherwise.

The solution of Equation 4.3 is generally approximated by Monte Carlo integration.

Suppose that g0, g1, . . . , gN, which are called gather samples, are random points dis-

tributed in scene surfaces uniformly. We have the following approximation:

Lo(p, ωo) ≈|U |N

N∑i=1

f(p, ωo, ωi)Lo(gi,−ωi)V (p, gi)(ωi · n)(−ωi · ni)

‖p − gi‖2

=N∑

i=1

f(p, ωo, ωi)T (p, gi)Lo(gi,−ωi) (4.4)

where

ni is the normal on gi.

|U | is the total surface area in scene.

ωi is the direction from p to gi.

T (p, gi) =

0, if p = gi

V (p, gi) ·|U |N

· (ωi · n)(−ωi · ni)

‖p − gi‖2, otherwise.

4.2 Diffuse direct-to-indirect transfer

Equation 4.4 shows that it is possible to treat light transfer as a linear system. The simplest

approach is assuming that all gather samples lie on perfect diffuse surfaces. This makes

Lo(g, ω) constant over ω and f(g, ωo, ωi) constant over ωo and ωi. Therefore we rewrite

Lo(g, ω) and f(g, ωo, ωi) as:

Lo(g, ω) = L(g)

f(g, ωo, ωi) = D(g)

and substitute p for another gather sample gj in Equation 4.4. Now it becomes:

L(gj) =N∑

i=1

D(gj)T (gj, gi)L(gi) (4.5)

This can be written in matrix form:

Ln = DLn−1

13

where

D is an N × N matrix and

Di,j = D(gj)T (gj, gi)

Ln =

Ln(g1)

Ln(g2)...

Ln(gN)

Ln(gi) is light intensity on gi by n-bounce indirect illumination.

In the relighting process, L0 is computed by direct illumination model. The light

intensity of gather samples with indirect illumination is obtained by:

L∗ = L0 + L1 + L2 + . . .

= L0 + DL0 + D2L0 + . . .

= (I + D + D2 + . . . )L0

= D∗L0

Gather samples are distributed in the precomputation stage and fixed in the relighting

process. Therefore D∗ is precomputable since D only depends on geometry and material

properties of gather samples. Generally it is sufficient for D∗ by including up to 4-bounce

transport matrix, i.e.

D∗ ≈ I +4∑

i=1

Di

After L∗ is computed, it is transferred to view samples defined in deep framebuffer.

Let v1, v2, . . . , vM be positions of shading points in the first layer of deep framebuffer.

The indirect illumination on view samples is computed by another matrix multiplication:

V = FL∗

where

F is an M × N matrix and

Fi,j = f(vi, ωvi→eye, ωvi→gj)T (vi, gj)

V =

Lo(v1, ωv1→eye)

Lo(v2, ωv2→eye)...

Lo(vM , ωvM→eye)

Finally V is added to the result rendered by direct illumination and now we have a

globally illuminated result.

14

4.3 Glossy direct-to-indirect transfer

In the previous section, we assume that all gather samples lie on perfect diffuse surfaces.

In this section we present a better approach to simplify Equation 4.4.

Suppose that all BRDFs on gather samples can be factorized as:

f(g, ωo, ωi) = ~h(g, ωo) · ~k(g, ωi)

and both ~h(g, ωo) and ~k(g, ωi) are δ-dimensional vectors. By BRDF factorization Equa-

tion 4.4 becomes:

Lo(gj, ωo) =N∑

i=1

[~h(gj, ωo) · ~k(gj, ωi)

]T (gj, gi)Lo(gi,−ωi)

= ~h(gj, ωo) ·

[N∑

i=1

T (gj, gi)~k(gj, ωi)Lo(gi,−ωi)

]

Let

Lo(gj, ωo) = ~h(gj, ωo) · ~λ(gj) (4.6)

~λ(gj) =N∑

i=1

T (gj, gi)~k(gj, ωi)Lo(gi,−ωi) (4.7)

and substitute Lo(gi,−ωi) in Equation 4.7 by Equation 4.6. Now we have:

~λ(gj) =N∑

i=1

T (gj, gi)~k(gj, ωi)[~h(gi,−ωi) · ~λ(gi)

]=

N∑i=1

[T (gj, gi)~k(gj, ωi) ⊗ ~h(gi,−ωi)

]~λ(gi) (4.8)

Here ⊗ denotes the outer product operator. Equation 4.8 is similar to Equation 4.5 except

that scalar multiplication is now replaced by matrix multiplication. It is possible to write

Equation 4.8 in matrix form:

Λn = GΛn−1

Here G is an (Nδ) × (Nδ) matrix composed by N × N smaller matrices:

G =

[G1,1] [G1,2] . . . [G1,N ]

[G2,1] [G2,2] . . . [G2,N ]...

... . . . ...

[GN,1] [GN,2] . . . [GN,N ]

Gi,j = T (gj, gi)~k(gj, ωi) ⊗ ~h(gi,−ωi)

15

And Λn is an (Nδ)-dimensional vector composed by ~λn(gi):

Λn =

~λn(g0)

~λn(g1)...

~λn(gN)

Although ~λn(gi) corresponds to Ln(gi) in the previous section, it does not mean light

intensity on gi. Light intensity is obtained by dot product with ~h(gi, ω):

Lno (gi, ω) = ~h(gi, ω) · ~λn(gi)

and here Lno (gi, ω) is light intensity on gi viewed from direction ω, by n-bounce indirect

illumination. ~λ0(gi) is computed by direct illumination model:

~λ0(gi) = Li(gi, ωi)~k(gi, ωi)

where Li(gi, ωi) is incoming radiance from light source directly.

Similar to the previous section, Λ∗ is computed and transferred to view samples:

Λ∗ = Λ0 + Λ1 + Λ2 + . . .

= Λ0 + GΛ0 + G2Λ0 + . . .

= (I + G + G2 + . . . )Λ0

= G∗Λ0

V = ΦΛ∗

However, Φ is now an M × (Nδ) matrix:

Φ =

[Φ1,1] [Φ1,2] . . . [Φ1,N ]

[Φ2,1] [Φ2,2] . . . [Φ2,N ]...

... . . . ...

[ΦM,1] [ΦM,2] . . . [ΦM,N ]

Φi,j =

[T (vi, gj)f(vi, ωvi→eye, ωvi→gj

)~h(gj, −ωvi→gj)]T

The definition of V here is as same as that in the previous section. It is added to the

image rendered by direct illumination as well.

4.4 BRDF factorization

In the beginning of the previous section, we have assumed that BRDFs on all gather

samples can be factorized as dot product of two vector functions. In this section the

details about BRDF factorization is presented.

16

We adopt the factorization method proposed by Kautz and McCool [11]. First both

incoming and outgoing directions of the factorized BRDF are discretized to a finite set:

Ω = ω1, ω2, . . . , ωK

and the reflectance matrix is formed:

M =

f(ω1, ω1) f(ω1, ω2) . . . f(ω1, ωK)

f(ω2, ω1) f(ω2, ω2) . . . f(ω2, ωK)...

... . . . ...

f(ωK , ω1) f(ωK , ω2) . . . f(ωK , ωK)

Depending on how specular f is, the rank of M varies from 1 to full rank. If f is a

perfect diffuse BRDF than f(ωo, ωi) is constant and M will have only 1 rank. The more

specular f is, the more rank M will have.

To exploit this feature, singular value decomposition (SVD) is applied to M :

M = UΣV T

where Σ is a diagonal matrix containing singular values in its diagonal. Most numerical

libraries, such as LAPACK [1] and GSL [6], would carefully order singular values such

that:

Σ1,1 ≥ Σ2,2 ≥ · · · ≥ ΣN,N ≥ 0

and we can approximate M by dropping smaller singular values. Let Σ′ be an N × N

diagonal matrix and

Σ′i,i =

0, if i > δ√

Σi,i, otherwise.

M ≈ U(Σ′Σ′)V T

= (UΣ′)(V Σ′)T

= (U ′)(V ′)T

Here U ′ is the matrix extracted from non-zero components in UΣ′, as well as V ′ from

V Σ′. The dimensions of U ′ and V ′ are both N × δ, and their row vectors correspond to~h(ω) and ~k(ω) respectively.

~h(ωi) = [U ′i ]

T

~k(ωj) =[V ′

j

]T

f(ωi, ωj) ≈ ~h(ωi) · ~k(ωj) (4.9)

17

(a)

~s~t

~nω

φ

θ

θ

φ

(b)

~nω

s

t

s

t

Figure 4.2: Projection from hemispherical directions to texture coordinates.

Equation 4.9 only factorize f for discrete directions. In the rendering process we need

to compute both ~h(ω) and ~k(ω) for arbitrary ω. Therefore we store U ′ and V ′ in texture

maps and interpolate ~h(ω) and ~k(ω) from nearby texels.

One simple approach to project spherical directions to texture coordinates is direct

mapping from (θ, φ) to (u, v):

u =2θ

π, v =

φ

2π

as illustrated in Figure 4.2 (a). This does not work well because we may have too much

samples for small θ but too few samples for θ near π/2.

Another approach, as shown in Figure 4.2 (b), projects ω to the tangent plane and

maps the projected direction to texture coordinate:

u =ω · ~s + 1

2, v =

ω · ~t + 1

2

This approach generally works better because the under-sampling problem is avoided.

Although some areas in the texture map cannot map to hemispherical directions and these

areas are wasted, this is tolerable because the texture map is usually very small. (32×32

for 812 sampled directions)

18

4.5 Limitations

In this chapter we proposed an extension to DTI that supports both diffuse and glossy

inter-reflections by BRDF factorization. However, there are some limitations in this algo-

rithm:

• Participating media such as smoke and fog is very common in cinematic effect, but

we assumed that there is no participating media.

• We assumed that all BRDFs on gather samples can be approximated by factoriza-

tion. This does not apply to highly specular BRDFs, and hence our algorithm does

not capture high frequency caustic.

• All factorized results of BRDFs must be stored in texture maps. In most scenes

there are only a handful of BRDFs and their factorized results can be shared among

gather samples. Even so, if there are BRDFs parameterized by texture maps, e.g.

roughness controlled by texture maps, the factorized results can not be shared. This

increases texture memory requirement vastly on GPUs.

We will leave these limitations for future work.

19

20

Chapter 5

Experiments and Results

5.1 Relignting system

We have implemented our relighting system and tested it on a Windows PC with an

Intel Pentium IV running at 3.4GHz, 2GB DDR SDRAM at 400MHz and an nVidia

GeForce 8800 GTX GPU. The resolution of all results in this section were set to 720×480.

We used Blender [3] as the reference production renderer, and hyper-threading was en-

abled when rendering reference images.

Our result is shown in Figure 5.1 (a). As a comparison, the reference image rendered

by Blender is shown in Figure 5.1 (c). Although there are slight differences between two

results, our relighting system gives a very convincible approximation as a preview.

The rendering times are listed in Table 5.1. Our system has a very short response

time after light parameters are modified. To ensure that our system scales well with large

number of light sources, we rendered the same scene with 16 light sources as shown in

Figure 5.1 (b). The performance is almost as fast as the scene with only two light sources.

Balls Bunny&Dragon

Triangle count 9,922 287,274

Preview with 2 lights 2∼3 msec 2∼4 msec

Preview with 16 lights 2∼3 msec 2∼4 msec

Final rendering with 2 lights 18.39 sec 107.20 sec

Final rendering with 16 lights 75.92 sec 340.16 sec

Table 5.1: Rendering time of our relighting system and Blender. The relighting preview

is generated by our system, and the final rendering result is generated by Blender.

21

(a) (b)

(c) (d)

(e) (f)

Figure 5.1: Test scene with one diffuse sphere and four specular ones.

(a). 2 lights rendered by our relighting system.

(b). 16 lights rendered by our relighting system.

(c). 2 lights rendered by Blender.

(d). 16 lights rendered by Blender.

(e). Difference map between (a) and (c), scaled by 2.

(f). Difference map between (b) and (d), scaled by 2.

22

(a) (b)

(c) (d)

(e) (f)

Figure 5.2: Test scene with complex models.

(a). 2 lights rendered by our relighting system.

(b). 16 lights rendered by our relighting system.

(c). 2 lights rendered by Blender.

(d). 16 lights rendered by Blender.

(e). Difference map between (a) and (c), scaled by 2.

(f). Difference map between (b) and (d), scaled by 2.

23

We also rendered another scene with complex models, as shown in Figure 5.2 (a).

They were also relighted by 16 light sources as shown in Figure 5.2 (b). As listed in

Table 5.1, our system maintains very short response time even with complex models and

large number of light sources.

With the help of difference maps in Figure 5.1 (e) (f) and Figure 5.2 (e) (f), there are

three major differences between our results and those rendered by Blender:

1. Generally anti-aliasing is enabled in final production, and we used 8 samples per

pixel in Blender. In our relighting system anti-aliasing was disabled. Although

anti-aliasing is possible by super-sampling the deep framebuffer, most users do not

expect it when previewing the rendering result.

2. The deep framebuffer used by our relighting system were generated by our pre-

processor, and the interpolation manner of normals differs from that by Blender.

Since specular reflections are very sensitive to normals, there are slight differences

in specular reflections.

3. To smooth away the edge of shadows, we implemented percentage closer filter-

ing [18] in our shader. However, it is not as accurate as ray traced shadows. Shadow

maps also cause some artifacts, such as “gaps” between walls in Figure 5.2 (c). This

is caused by shadow map bias which is necessary for shadow map implementations.

5.2 Glossy direct-to-indirect transfer

To validate our algorithm, we have implemented both diffuse and glossy DTI as surface

integrator plug-ins for PBRT [16]. All results in this section were generated on a Linux

PC with an AMD Athlon 64 X2 running at 2.2GHz and 2GB DDR SDRAM at 400MHz.

The resolution of all results in this section were set to 512×512. As the reference ground

truth, we used the result rendered by path tracing plug-in in PBRT and set the number of

samples per pixel to 1024 for high quality.

As shown in Figure 5.3, both diffuse DTI and glossy DTI approximate global illu-

mination reasonably. Note that both two results are noisy. This is caused by insufficient

gather samples. Since gather samples are distributed on all surfaces in scene, a typical

number is 65,536 samples as indicated by Hasan et al. [8]. This makes the transfer matrix

very large and matrix compression is essential. Our implementation, however, does not

include matrix compression and only 2,304 gather samples are used.

In order to compare the difference between diffuse and glossy inter-reflections, we

stripped off the diffuse term of all BRDFs and subtracted the direct illuminated result

from the global illuminated results. Finally the intensity of these images is scaled by 4

24

(a) (b)

(c) (d)

Figure 5.3: Cornell box by different illumination algorithms.

(a). Direct illumination

(b). Path tracing

(c). Diffuse direct-to-indirect transfer

(d). Glossy direct-to-indirect transfer

25

Cornell box Buddha

Path tracing 5321 sec 4700 sec

Diffuse DTI 1001 sec 913 sec

Glossy DTI 2494 sec 2308 sec

Table 5.2: Rendering times of difference global illumination algorithms.

and shown in Figure 5.4. Obviously our algorithm generates a better approximation than

diffuse DTI when dealing with indirect illumination caused by glossy materials. Note that

corners between walls are generally darker in results rendered by both diffuse DTI and

glossy DTI. This is also caused by insufficiency of gather samples.

Figure 5.5 shows another test scene. Apparently diffuse DTI gives a result too bright,

and the blurred reflection by Buddha does not appear. There is blurred reflection in the

result by glossy DTI, but it is not as clear as that in result by path tracing because of

insufficient gather samples.

Although the rendering times are not relevant, they are listed in Table 5.2. Since our

algorithm is based on DTI, we expect interactive rendering time after matrix compression

is applied and the whole algorithm is implemented on GPUs.

26

(a)

(b) (c)

Figure 5.4: Indirect illumination by glossy-only BRDFs, scaled by 4.

(a). Path tracing

(b). Diffuse direct-to-indirect transfer

(c). Glossy direct-to-indirect transfer

27

(a) (b)

(c) (d)

Figure 5.5: Buddha by different illumination algorithms.

(a). Direct illumination

(b). Path tracing

(c). Diffuse direct-to-indirect transfer

(d). Glossy direct-to-indirect transfer

28

Chapter 6

Conclusion and Future Work

In this thesis we have presented an interactive relighting system which provides fast feed-

back for lighting artists. We cache shading parameters in deep framebuffer to reduce

scene complexity and guarantee frame rate by shaders running on GPUs. Shadows are

calculated by shadow maps, and we also provide specular reflections by layered deep

framebuffer.

In the second part we discussed global illumination for relighting systems based on

gather samples. We proposed a new approach based on DTI and extended light transport

path to both diffuse and glossy inter-reflections by BRDF factorization.

Although we have tested our algorithm with a software renderer, it is not practical

until it is ported and validated on GPUs. It is also essential to compress the transfer

matrix for both large number of gather samples and fast matrix multiplication on GPUs.

This remains as the primary task of our future work. There are also limitations in our

algorithm since we have assumed that there is no participating media and all BRDFs can

be factorized and stored. We expect that these limitations can be relaxed in the near future

by novel solutions.

29

30

Bibliography

[1] E. Anderson et al. LAPACK Users’ Guide. Society for Industrial and Applied Math-

ematics Philadelphia, third edition, 1999.

[2] T. Annen, J. Kautz, F. Durand, and H.-P. Seidel. Spherical harmonic gradients for

mid-range illumination. In Proceedings of Eurographics Symposium on Rendering,

pages 331–336, 2004.

[3] Blender, http://www.blender.org/.

[4] N. Briere and P. Poulin. Hierarchical view-dependent structures for interactive scene

manipulation. In Proceedings of ACM SIGGRAPH, pages 83–90, 1996.

[5] F. Crow. Shadows algorithms for computers graphics. In Proceedings of ACM

SIGGRAPH, 1977.

[6] M. Galassi, J. Davies, J. Theiler, B. Gough, G. Jungman, M. Booth, and F. Rossi.

GNU Scientific Library Reference Manual - Revised Second Edition (v.18), 2006.

[7] R. Gershbein and P. Hanrahan. A fast relighting engine for interactive cinematic

lighting design. In Proceedings of SIGGRAPH, pages 353–358, 2000.

[8] M. Hasan, F. Pellacini, and K. Bala. Direct-to-indirect transfer for cinematic relight-

ing. In Proceedings of ACM SIGGRAPH, pages 1089–1097, 2006.

[9] H. W. Jensen. Global illumination using photon maps. In Proceedings of the Euro-

graphics workshop on Rendering techniques, pages 21–30, 1996.

[10] J. T. Kajiya. The rendering equation. In Proceedings of ACM SIGGRAPH, pages

143–150, 1986.

[11] J. Kautz and M. D. McCool. Interactive rendering with arbitrary brdfs using separa-

ble approximations. In Proceedings of the 10th Eurographics Rendering Workshop,

pages 281–292, 1999.

31

[12] X. Liu, P.-P. Sloan, H.-Y. Shum, and J. Snyder. All-frequency precomputed radiance

transfer for glossy objects. In Proceedings of Eurographics Symposium on Render-

ing, pages 337–344, 2004.

[13] R. Ng, R. Ramamoorthi, and P. Hanrahan. All-frequency shadows using non-linear

wavelet lighting approximation. In Proceedings of ACM SIGGRAPH, pages 376–

381, 2003.

[14] R. Ng, R. Ramamoorthi, and P. Hanrahan. Triple product wavelet integrals for all-

frequency relighting. In Proceedings of ACM SIGGRAPH, pages 477–487, 2004.

[15] F. Pellacini, K. Vidimce, A. Lefohn, A. Mohr, M. Leone, and J. Warren. Lpics:

a hybrid hardware-accelerated relighting engine for computer cinematography. In

Proceedings of ACM SIGGRAPH, pages 464–470, 2005.

[16] M. Pharr and G. Humphreys. Physically Based Rendering: From Theory to Imple-

mentation. Elsevier/Morgan Kaufmann, 2004.

[17] J. Ragan-Kelley, C. Kilpatrick, B. W. Smith, D. Epps, P. Green, C. Hery, and F. Du-

rand. The lightspeed automatic interactive lighting preview system. To appear in

Proceedings of ACM SIGGRAPH, 2007.

[18] W. T. Reeves, D. H. Salesin, and R. L. Cook. Rendering antialiased shadows with

depth maps. In Proceedings of ACM SIGGRAPH, pages 283–291, 1987.

[19] T. Saito and T. Takahashi. Comprehensible rendering of 3-d shapes. In Proceedings

of SIGGRAPH, pages 197–206, 1990.

[20] C. H. Sequin and E. K. Smyrl. Parameterized ray tracing. In Proceedings of SIG-

GRAPH, pages 307–314, 1989.

[21] P.-P. Sloan, J. Kautz, and J. Snyder. Precomputed radiance transfer for real-time

rendering in dynamic, low-frequency lighting environments. In Proceedings of ACM

SIGGRAPH, pages 527–536, 2002.

[22] R. Wang, J. Tran, and D. Luebke. All-frequency relighting of non-diffuse objects

using separable brdf approximation. In Proceedings of Eurographics Symposium on

Rendering, pages 345–354, 2004.

[23] G. J. Ward, F. M. Rubinstein, and R. D. Clear. A ray tracing solution for diffuse

interreflection. In Proceedings of ACM SIGGRAPH, pages 85–92, 1988.

[24] T. Whitted. An improved illumination model for shaded display. Communications

of the ACM, pages 343–349, 1980.

32

[25] L. Williams. Casting curved shadows on curved surfaces. In Proceedings of ACM

SIGGRAPH, pages 270–274, 1978.

33