The candidate confirms that the work submitted is their own and the appropriate credit has been given where reference has been made to the work of others.

I understand that failure to attribute material which is obtained from another source may be considered as plagiarism.

(Signature of student)_________________________________

Realistic Rendering Of Light And Materials Using The GPU

Ben Wells

BSc. Computing

2008/2009

Summary

This report presents an in depth review into representing the interaction of light with a number of materials

using programmable computer graphics hardware. It also documents the path that was taken to produce

several videos that show scenes of screen captured computer graphics that were rendered in real time (25 -

60 frames per second) based on physically realistic models of light and materials. The report also shows the

battle between physical realism of real-time computer graphics against the need for fast rendering speeds and

documents the decisions and sacrifices made to achieve real time, physically based computer graphics. The

report then concludes with an evaluation of the scenes and the mathematical models chosen to build them.

i

Acknowledgements

First and foremost I would like to thank my project supervisor Ken Brodlie for his invaluable comments and

constructive criticism and to Mum, Dad, Rich and Hannah for supporting me, not just in this project, but

throughout my four years at Leeds University. I couldn't have done it without you guys!

ii

Contents

1 Introduction 1 1.1 Overview ............................................................................................................................................ 1

1.2 Aim .................................................................................................................................................... 2

1.3 Objectives .......................................................................................................................................... 2

1.4 Requirements ..................................................................................................................................... 3

1.4.1 Minimum Requirements ........................................................................................................... 3

1.4.2 Extending the Minimum Requirements .................................................................................... 3

1.5 Deliverables ....................................................................................................................................... 4

1.6 Schedule ............................................................................................................................................. 4

1.7 Similar Projects .................................................................................................................................. 5

1.8 Summary ............................................................................................................................................ 6

2 Research – Modelling Light/Material Interaction 7 2.1 Introduction ........................................................................................................................................ 7

2.2 The Rendering Equation .................................................................................................................... 8

2.3 Phong Shading ................................................................................................................................... 10

2.3.1 Ambient Term ............................................................................................................................ 10

2.3.2 Diffuse Term .............................................................................................................................. 10

2.3.3 Specular Term ............................................................................................................................ 11

2.3.4 The Phong Reflection Model ..................................................................................................... 11

iii

2.3.5 Phong Interpolation .................................................................................................................. 12

2.3.5.1 Representing a 3D Model in Computer Graphics .......................................................... 12

2.3.5.2 The Theory of Phong Interpolation ............................................................................... 13

2.4 Spherical Harmonic Lighting ............................................................................................................ 14

2.4.1 The Associated Legendre Polynomials ..................................................................................... 14

2.4.2 Spherical Harmonics ................................................................................................................. 15

2.4.3 Spherical Harmonics in Practice ............................................................................................... 15

2.4.4 Weaknesses of Spherical Harmonic Lighting ........................................................................... 18

2.5 Summary ............................................................................................................................................ 18

3 Research - Tools 19 3.1 Introduction ........................................................................................................................................ 19

3.2 The Rendering Pipeline ..................................................................................................................... 20

3.3 The GPU and the Fixed and Programmable Rendering Pipelines ..................................................... 20

3.4 Direct3D, OpenGL and Shaders ........................................................................................................ 21

3.5 HLSL and GLSL ................................................................................................................................ 22

3.6 Direct3D Effects Framework ............................................................................................................. 23

3.7 SIMD ................................................................................................................................................. 25

3.8 Summary ............................................................................................................................................ 26

4 Design and Implementation - Methodology 27 4.1 Overview of Methodologies .............................................................................................................. 27

4.2 Custom Methodology ........................................................................................................................ 28

5 Design and Implementation – The Rendering Engine 30 5.1 Introduction ........................................................................................................................................ 30

5.2 Design Decisions and Justifications .................................................................................................. 30

5.2.1 Graphics API ............................................................................................................................. 30

5.2.2 Programming Language ............................................................................................................ 31

5.3 Manager Modules .............................................................................................................................. 31

5.4 Summary ............................................................................................................................................ 32

6 Design & Implementation – The Shaders and Techniques 33 6.1 Introduction ........................................................................................................................................ 33

6.2 The Water Scenes ............................................................................................................................... 34

6.2.1 Geometric Water Movement ..................................................................................................... 34

6.2.2 Light/Water Interaction ............................................................................................................. 35

6.2.2.1 Calculating the Normal of Each Vertex ........................................................................ 36

6.2.2.2 Modelling Refraction .................................................................................................... 37

6.3 The Glass Mummy ............................................................................................................................. 38

6.3.1 Using Snell's Law To Model Refraction ................................................................................... 39

iv

6.3.2 Modelling Reflection ................................................................................................................ 39

6.3.3 Combining Refraction and Reflection ...................................................................................... 40

6.4 The Wax Models ................................................................................................................................ 41

6.5 Summary ............................................................................................................................................ 43

7 Evaluation 44 7.1 Introduction ........................................................................................................................................ 44

7.2 Evaluating the Objectives .................................................................................................................. 45

7.3 Public Evaluation Method ................................................................................................................. 45

7.4 Evaluation Results and Improvements .............................................................................................. 47

7.5 Conclusion and Graphics Turing Test ................................................................................................ 49

Bibliography 50 Appendix A 53

v

Chapter 1

Introduction

1.1 Overview

Computer graphics has been the subject of much academic research over the years. It can be argued

that through the popularisation of computer games and computer generated motion pictures, the product of

50+ years research has now truly found its place; amongst the mainstream entertainment industry. The

British video games industry now contributes £1 billion to GDP of Great Britain and this in turn fuels

academic research in the area of real time computer graphics.

Recent innovations in computer graphics hardware (described in chapter 3) along with the ongoing

quantitative increase in affordable computer memory and CPU power have allowed computer graphics

programmers in the computer games industry to use sophisticated and complex physically based models of

light and materials to increase the realism of their graphical scenes. Algorithms that were only available to

non real-time computer graphics applications are now finding their place in real-time applications. The

envelope is constantly being pushed with every computer game released.

1

This project aims to explore some of these realistic light and material models used in modern

computer games and evaluate to what extent these models affect the realism of computer graphics. The

project will also implement some real time graphical scenes that use physically based models of light and

materials using programmable graphics hardware.

1.2 Aim

This project aims to build realistic models of light and its interaction with materials such as glass and

water using programmable graphics hardware. Furthermore, the models should strive to operate in real-time,

meaning a frame should be rendered at least once every 0.04 seconds (25 frames per second) but ideally as

much as one frame every 0.016 seconds (60 frames per second) on a standard home PC. The reason for a

real-time constraint is because it allows the pursuit of real-time graphics algorithms; an area of particular

interest in the computer games industry.

The project then aims to evaluate the realism of the models. This will be in the form of a public

evaluation whereby the models will be judged on exactly how true to life they appear. If they are found to be

noticeably different from reality, the evaluation should provide answers to why they are different and what

aspects of the visualisation make them different.

Overall, the aim is to provide a greater understanding about what aspects of graphics make them

appear true to life and what aspects make them appear “fake”, and how affective physically based models of

light and materials are at conveying realism in computer graphics on modern computer hardware.

This project also aims to propose a computer graphics equivalent to the Turing Test (Turing, 1950). The

Turing Test aims to test whether or not a computer could imitate a human. This project would propose a

similar test, but would evaluate whether or not a computer could imitate reality in a visual sense. This project

aims to lay down the rules for the test and conditions with regards to how to pass it.

1.3 Objectives

Here is a list of milestones that must be met in order to achieve the project aims.

1. A fluency with the concepts of programming graphics hardware must be developed. This involves

acquiring the knowledge to compile, link, and execute programs on graphics hardware using a

computer graphics application programming interface (API) such as Direct3D or OpenGL.

2. The project will involve the study of an API specific programming language that can be compiled

and executed on the graphics hardware. A low level understanding should be developed. The reason

for this is because the programs need to be as efficient as possible in order for them to execute

scenes in real time.

3. A system must be built that easily allows the user to compile, link and execute programs on graphics

hardware. The system should be highly optimised to allow for a fast program execution; vital for

2

real-time graphics applications.

4. Research must be done regarding the mathematical models of light and its interaction with materials.

5. The project should provide detailed graphical scenes that are implemented as efficiently as possible

using algorithms that allow for the visual appearance of complexity without using up all the

processing power of the graphics card (a common goal in computer graphics).

6. The implementation section of the project should culminate with the production of several scenes

that will be evaluated by the public using an evaluation platform such as a web site.

7. When the public evaluate the scenes, they will be asked questions to do with the reality of what they

are viewing. These questions are important because the answers the public provide must

communicate exactly what aspects of the scenes make them appear real/unreal. Therefore, the

questions should be carefully thought out in order to get the most useful feedback.

8. The public's response to the scenes will then be evaluated and conclusions will be made about what

aspects of the scenes appear real and fake and what improvements can be made to the scenes.

9. Using the results of the public evaluation and the experience gained throughout this project, the rules

for the graphics equivalent to the Turing Test will be published on the WWW.

1.4 Requirements

1.4.1 Minimum Requirements

Here is a list of the minimum requirements for the project.

1. Produce one scene that models a real-life scenario.

2. Provide a platform upon which the public can evaluate that scene.

3. Provide evidence of how realistic the scene is using the feedback from the public evaluation.

4. Propose the graphics Turing Test, explaining the rules and pass requirements.

1.4.2 Extending the Minimum Requirements

Meeting these minimum requirements should provide an adequate solution to the problem, but

extending these requirements will allow for a greater understanding of what makes graphics appear

real/unreal. To surpass the minimum requirements the project should provide a range of scenes that model

several different real life situations. Such scenes could be:

● Flowing water.

● A surface such as cobbles or tarmac.

● A Glass object.

● Refraction effect of a raindrop on glass.

3

1.5 Deliverables

The main deliverable of the project will be in the form of a public website made available on the World Wide

Web at http://www.benwells3d.com:

● Containing several video files of the scenes produced.

● Containing the evaluation test undertaken by the public.

● Containing a downloadable document proposing the graphics Turing Test

● Containing the results and conclusions of the project.

This project report is also a deliverable of this project because it provides and in depth explanation into the

project as a whole and the mathematical models used to model the scenes.

1.6 Schedule

Here is the schedule for the project.

1. [September until Christmas] Read up on the physics of light and its interaction with materials,

particularly how it has been implemented with computer graphics in the past.

2. [August until Christmas] Read up on programming graphics on programmable graphics hardware.

3. [September until Christmas] Build a system that allows the user to easily load, compile, link and

execute programs on graphics hardware.

4. [September until Christmas] Decide exactly on what real life situations the project should model.

These decisions should be based on the background reading.

5. [September until February] Model and render the chosen scenes.

6. [November until December 12th] Write up the mid project report

7. [Christmas until January] Exams and revision.

8. [February] Evaluate the set of simulations. This will be done by rendering out the graphical scenes to

video files. Then, a basic web application will be set up as a platform for the public to evaluate the

videos.

9. [April – May] Write a document proposing the graphics Turing Test.

10. [April –May] Publish on-line.

11. [Christmas until May] Write the report.

4

1.7 Similar Projects

A great deal of advanced computer graphics tries to mimic reality in some way or another. Indeed, it

has been the goal of many film makers to incorporate computer graphics into their films in such a way as to

make indistinguishable what is real on the screen and what is not. Take a film series such as Lord of the

Rings for example, where computer graphics were used to present the raging fires of Mount Doom and the

lush forests of Rivendell. The key differences here, of course, are that these scenes were not real-time and the

Lord of the Rings technical team had millions of dollars to spend. However, the aims for this project are not

to create the scenes from Mount Doom, but to present on screen a single material that looks real.

However, real-time scenes that mimic reality have been attempted, such projects as The Human

Head (figure 1) (d'Eon and Luebke , 2007) try to realistically render human skin. Also, features from the new

Crytek graphics engine, CryEngine2 (figure 2), provide the user with the ability produce amazingly realistic

scenes in real time using many complex materials and hugely complex lighting algorithms; see Crytek

(2008) for more details.

Figure 1. Screen shot from Nvidia's human head demo.

5

Figure 2. Screen shot from CryEngine2, Crytek's new graphics engine.

1.8 Summary

This chapter has laid out the fundamentals of this project. In particular, a high level view into the

themes that this project will explore, how they will be explored, and what will result has been discussed.

Some similar projects that have looked into physically realistic rendering have also been discussed.

6

Chapter 2

Research – Modelling Light/Material Interaction

2.1 Introduction

As stated in the project aims and minimum requirements, this project aims to build a realistic model

of light and its interaction with a material. This model can then be used to render a graphical scene that

approaches a look similar to reality. This project can be taken as a direct advancement from the computer

graphics modules that were taught as part of the Computing undergraduate degree program at the University

of Leeds. These modules, in particular GI21 (Introduction To Computer Graphics), served as an introduction

into modelling light and surface reflectance in real time computer graphics. The main bulk of the research

undertaken in this project follows directly from the fundamental concepts taught in these modules.

The first section of this chapter will describe the current gold standard of light/material interaction

and address why the gold standard is currently only achievable in non real-time using modern graphics

7

hardware. Next, some of the more simple lighting/material mathematical models will be presented that

attempt to provide a simple approximation to the gold standard allowing for real-time applications.

Following this, a complex model will be presented that matches very closely to the gold standard.

2.2 The Rendering Equation

The rendering equation in computer graphics was introduced simultaneously by David Immel et al

(1986) and James Kajiya (1986). It is an integral equation which approximates the amount of radiance

leaving a single point on a surface as the sum of emitted and reflected radiance from that point. The two

models discussed in sections 2.3 and 2.4 aim to solve this equation. The rendering equation may be given as:

(1)

where,

L x ,o is the total radiance from point x along vector o .

Le x ,o is the radiance emitted from point x along vector o .

f r x ,io is the reflectance function at point x . This is known as the Bidirectional Reflectance

Distribution Function (BRDF) and describes how light is reflected from an opaque surface. Many BRDFs

exist including Cook and Torrance (1981), Nayar and Oren (1995) and Minnaert (1941) and are often used to

simulate different materials such as metal, clay and velvet respectively.

L x ' ,i is the radiance arriving from another point, x ' (located either on the same or a different

object) along vector i . This can be viewed as light that has already been reflected off another local

object and is not directly from a light source.

G x , x ' is the geometric relationship between point x and x ' and is responsible for scaling

V x , x ' (defined below). The geometric term describes things such as the distance between points x

and x ' as well as any other geometric information deemed necessary.

V x , x ' is an occlusion test; returns 1 if x can see x ' , 0 otherwise.

The integral term, S, defines this integral over a hemisphere surrounding position x and can be

viewed as the distribution of incoming light towards point x . This light is the combination of all radiance

arriving along every i existing in the hemisphere including the inter-reflected light arriving from other

objects.

Due to the inter-reflected light arriving along i it is intuitive to see that the rendering equation

must be applied recursively. The recursive nature of the rendering equation makes it very real-time

8

L x ,o=Le x ,o∫S

f rx ,ioL x ' ,iG x , x ' V x , x ' di

unfriendly. This is because every point on the object must recursively apply the rendering equation to gather

the reflected light from all other visible points in the scene.

The rendering equation can be solved using a backward ray tracer. This attempts to solve the

rendering equation by projecting rays from the camera and reflecting them around the scene where each

point of reflection casts a hemisphere of rays back into the scene. The ray tracer is recursive by nature and

hence very slow. Figure 3 shows the results of rendering a scene using a backward ray tracer.

Figure 3. The results of a ray tracer.

http://en.wikipedia.org/wiki/File:Glasses_800_edit.png

Figure 3 shows how true to life scenes can look if the rendering equation can be solved. However

figure 4 shows my implementation of a ray tracer with a scene rendered on a Dell Inspiron 6400 laptop. The

scene models reflection and refraction very nicely but took 34 minutes to render; definitely not real time!

Figure 4. The results of a ray traced scene on a standard laptop. 34 minutes rendering time.

9

Whilst the rendering equation may be the way to go in terms of realism, solving it using backwards

ray tracing is far too slow to provide real-time results using a standard PC or laptop. However, research has

uncovered other methods that are applicable for real-time scenarios. Whilst rendering speed is increased

using these methods, physical realism is compromised. This is inevitable because assumptions and

simplifications have to be made to the rendering equation in order to achieve real time rendering. The

following sections explains two of these methods. It is also worth noticing that the rendering equation does

not model light scattering within an object before leaving the object towards the eye, often seen in

translucent objects, a technique for achieving this is discussed in section 6.4.

2.3 Phong Shading

Phong Shading (Phong, 1975) is perhaps the most widely used method for modelling light/material

interaction in real-time scenes and combines the Phong Reflection Model; a simplification of the rendering

equation, with a method for estimating pixel colours by interpolating surface normals across an object. To

better understand Phong shading we must first break light into its component parts (as modelled by the

Phong Reflection Model).

2.3.1 Ambient Term

In the Phong Reflection Model ambient light can be thought of as light that is emitted from the

object itself. Ambient light is emitted uniformly in all directions within the hemisphere surrounding a single

point and therefore it can be seen as a simplification to the term Le x ,o given in equation 1 where

o drops out of the term to yield a constant value which is donated k a i a ; also known as the ambient

term. Here ia represents the intensity of the ambient light and k a represents the amount of ambient light

being emitted from the point in question.

Another way to think of the ambient term is that it compensates for inter-reflected light. However,

the Phong Reflection does this by blindly adding light into the scene. This, therefore, cannot be directly

mapped to the rendering equation because the portion of inter-reflected light at a point must take into account

all points visible to the point in question and their reflected light. Due to this reason I believe that it is more

intuitive to visualise the ambient term as the light that is emitted from a point to compensate for the fact that

the Phong Reflection model does not correctly calculate inter-reflected light at that point.

2.3.2 Diffuse Term

Diffuse light can be thought of as the true colour of the matte component of the material. It is best

conveyed through an example. The sun's light can be thought of as white light. The visible part of the white

light can be thought of the sum of all wavelengths present in the visible part of the electromagnetic spectrum.

Upon hitting a blue object, all but the blue light within the white light is absorbed by the object. The

10

reflected blue light hits the retina in the eye and we see the blue object. This blue light is labelled diffuse and

is modelled in the Phong reflection model as k d L⋅N i d ; also known as the diffuse term. Here k d is

the amount of diffuse light reflected from the point in question. L is the unit vector from the point to the

light. N is the unit normal of the surface at the point and id is the intensity of the diffuse light. N⋅L

is also know as Lambert's Cosine Law which states that the radiant intensity observed from a surface is

directly proportional to the cosine of the angle between the light's line of sight and the surface normal.

2.3.3 Specular Term

Specular light models the perfect reflection of light upon interaction with a material. Specular

reflection can be found on shiny materials and can be explained because many materials have a thin clear

layer above the coloured “diffuse” layer. This thin layer may reflect the entire visible spectrum of light

present when light is incident at the correct angle given the viewer's position and the position of the light

source. Specular light is modelled as k sR⋅V i s where k s is the amount of specular light reflected,

R is the vector between the point in question and the light source reflected about the normal to the point

in question, V is the vector from the point in question to the eye, is the shininess of the material and

i s is the intensity of the specular light (usually the colour of the light being emitted from the light source).

2.3.4 The Phong Reflection Model

The ambient, diffuse and specular terms defined above are combined to give the Phong Reflection

Model; given in equation 2.

(2)

The first thing to notice is the similarity between equation 2 and equation 1. The integral in equation

1 has been converted to a summation over the number of lights in the scene. This is because the Phong

Reflection Model treats lights as single points of zero size. Therefore the total energy emitted by the light can

be captured in one vector; L in this case. The removal of the integral is a massive optimisation because we

can now evaluate the radiance using far less computation; we now only need to calculate the diffuse and

specular terms once per light source for each point in the scene.

Notice how the occlusion test of equation 1, V x , x ' , has also been dropped from the equation.

This optimisation drastically decreases the realism of the scene because any shadowing information that may

be present has now been lost but rendering speed is increased because we do not have to check if a point is

occluded; Phong Shading alone cannot model occlusion.

Also notice how inter-reflectance has completely dropped from the equation leaving only the BRDF

where the diffuse and specular term form the BRDF term f r x ,io given in equation 1. This lack

of inter-reflectance greatly decreases the physical realism of a scene but provides a huge improvement in

11

I p=k a I a∑lights

k d N⋅Li dk sR⋅V a i s

rendering speed because the recursive element of the rendering equation has disappeared, therefore, when

equation 2 is used in computer graphics, it can be used to light objects in real-time. Figure 5 gives a visual

representation of the different parts of the Phong Reflection Model and how they combine with vertex

normal interpolation (explained in the next section) to create the final result.

Figure 5. Ambient, Diffuse and Specular terms.

http://en.wikipedia.org/wiki/File:Phong_components_version_4.png

2.3.5 Phong Interpolation

Phong Interpolation ( Phong, 1975) works together with Phong reflection to produce Phong Shading.

Whilst the interpolation techniques applied in Phong shading are not based on physical reality they do offer

an increased level of realism to the scene. To better understand what Phong Interpolation is we need to

introduce how data is represented in computer graphics to form 3D models. After this the theory of Phong

Interpolation will be discussed.

2.3.5.1 Representing a 3D Model in Computer Graphics

A common method of representing a 3D model in computer graphics is to use a data structure called

a vertex (Luna, 2006). The vertex contains information about a single point in 3D space. The reader may find

it helpful to imagine that the vertex exists at a position in 3D space and contains information that can be used

by lighting equations. The vertex can contain any information deemed necessary but it is common for a

vertex to contain the following:

● Vertex Position – The position of the vertex in 3D space with respect to a local coordinate system.

● Adjacency Information – The positions of the vertices adjacent to and connected via an edge to the

vertex (see figure 6).

● Vertex Normal – The vector formed by taking the average of the unit vectors perpendicular to the

faces adjacent to the vertex (see figure 6).

12

2.3.5.2 The Theory of Phong InterpolationGiven the normal information of two adjacent vertices connected by a single edge, Phong

Interpolation can estimate the normals for positions along that edge. One way this can be achieved is to use

equation 3 where N left and N right are the vertex normals of two adjacent vertices and p is a scalar

from 0 to 1.

N new= p N left1− pN right (3)

With this newly calculated normal information, the Phong Reflection Model (equation 2) can, in

theory, be applied at as many points on the surface of the model as we wish. When applied in computer

graphics, the only constraint is the physical width of a pixel on the computer monitor, hence this method is

also known as per pixel lighting.

Applying Phong Interpolation to a model gives the illusion of a very smooth surface and when

combined with Phong Reflection it gives the appearance of smooth plastic. However, Phong Interpolation

does decrease the rendering speed due to the increase in the number of times we need to apply the Phong

Reflection Model. This is seen as a compromise worth taking because of the visual improvement to the

scene; as figure 7 shows.

Figure 7. The same model rendered without and with Phong Shading.

http://en.wikipedia.org/wiki/File:Phong-shading-sample.jpg

13

Figure 6. Notice how the 3 vertices (circles) form 3 edges. Notice how V (the light coloured circle) has the normal N, taken as the average of vectors N Face 1 and N Face 2.

NV

N NFace 1Face 2

2.4 Spherical Harmonic Lighting

In section 2.3 a heavily simplified solution to the rendering equation was discussed. This section

introduces another method that can solve the entire rendering equation and allow for a real-time scene. First

introduced by Kautz et al (2002) and further explained by Green (2003), this method is used in the computer

games industry (Bungie, 2009) but does not come without its weaknesses. The theory behind spherical

harmonic lighting lies in orthonormal basis functions and representing a low frequency approximation of a

lighting function and the lighting function's interaction with a model using compression techniques

analogous to the techniques involving the Fourier Transform and the Discrete Cosine Transform.

This report only scratches the surface of spherical harmonics in computer graphics, readers should

refer to Green (2003) for a more in depth discussion.

2.4.1 The Associated Legendre Polynomials

The associated Legendre polynomials lie at the heart of spherical harmonic lighting. They define a

set of orthonormal polynomials often given as P defined over the range [-1, 1] and take 2 arguments,

namely the band index, often represented with l which can be any positive integer value from 0, and the

integer term m valued in the range [0, l ]. Together, the arguments l and m break the polynomials

into bands of functions, hence the name “band index” for l .

One way of evaluating the bands of associated Legendre polynomials, ideal for a computer program,

is to use the set of recurrence relations given in equations 4, 5 and 6. These recurrence relations allow a

systematic approach to evaluating P lm x for a given l and m and x by generating the current

polynomial from previous polynomials in the series.

Pmm x=−1m2m−1!!1− x2

m2 (4)

Pm1m x =x 2 m1Pm

m (5)

l−mP lm x= x 2 l−1P l−1

m −lm−1P l−2m (6)

Note that x!! is the double factorial function which, as (2m–1) is always odd, returns the product of

all odd integers less than or equal to x

It is intuitive to see that equation 6 can be evaluated using the results of applying equation 4 and then

equation 5. Another thing to note about equations 4, 5 and 6 is that they return real values, as opposed to

complex values that are returned when evaluating, not the associated Legendre polynomials, but the ordinary

Legendre polynomials; it is important not to confuse the two.

14

2.4.2 Spherical HarmonicsDue to the nature of the rendering equation, we need to represent the associated Legendre

polynomials calculated using equations 4, 5 and 6 over the 2D surface of a sphere (remember that the

rendering equation evaluates over an entire hemisphere). P lm x is essentially a 1D function of x , to

be able to represent this value using a polar coordinate system defined over a unit sphere we need to convert

the value into spherical coordinates. This is shown in equations 7 and 8 and evaluates the real spherical

harmonic function, y lm , where and are the polar coordinates.

y lm ,={ 2 K l

m cos mP lm cos , m0

2 K l−m cos −m P l

−mcos , m0K l

0 cos , m=0 (7)

where,

K lm= 2l1l−∣m∣!

4l∣m∣! (8)

In equation 7 and 8, m is declared differently from the associated Legendre polynomials, now m falls

in the range −l to l .

2.4.3 Spherical Harmonics in Practice

This section will address how we apply what was discussed in sections 2.4.1 and 2.4.2 where we

represented the associated Legendre polynomials over a unit sphere. The algorithm used to solve the

rendering equation is pre-computed off-line before rendering begins and the results are stored in a file, the

algorithm is as follows:

1. Uniformly sample 10,000 points over a unit sphere using a sampling scheme such as stratified

sampling (Green 2003).

2. Calculate the spherical harmonic functions for each sample using equations 7 and 8 for every value

of m in the range −l to l for every l from 0 to lmax . I chose l=4 for my

implementation and therefore was left with 16 numbers per sample, these numbers are known as

spherical harmonic coefficients and provide a new basis that enables us to project arbitrary lighting

functions onto a spherical harmonic basis.

3. Define a lighting function (I used the lighting function given in equation 9) and project this lighting

function into a spherical harmonic basis (see equation 10) using the spherical harmonic coefficients

calculated in step 2. This leaves 16 coefficients that represent a low frequency version of the lighting

function in terms of a spherical harmonic basis.

15

light ,=max 0, 14

sin sin max0, 14

sin−cos (9)

c i=4N ∑

j=1

N

light j , jS i j , j (10)

where,

c i is one of the 16 coefficients; number i .

N is the number of samples; 10,000 in this case.

j and j are the polar coordinates of sample j .

S i is one of the 16 coefficients; number i of the sample given by j and j .

4. For each sample S taken in step 1, for each vertex normal N on the model, calculate S⋅Nand if the resulting scalar is greater than 0, determine if the vertex position is occluded by another

part of the scene using a standard ray tracing technique. If it is not occluded then project the result of

S⋅N onto a spherical harmonic basis the same way as the lighting function in step 3.

5. Because l=4 the projection done in steps 3 and 4 leaves 16 coefficients that represent a low

frequency approximation to the light function and 16 coefficients per vertex that represents an

encoded version of the reflectance function for that vertex. Now, due to the guaranteed fact that the

spherical harmonic functions are orthonormal because of the associated Legendre polynomials the

solution to the integral of the rendering equation (equation 1) at any given vertex is the dot product

of the 16 coefficients obtained in steps 3 and step 4. As shown in equation 11.

∫S

f r x ,ioV x , x ' di≈∑i=0

16

c i r i (11)

where,

c i is the ith coefficient of the 16 coefficients that represents the light function in terms of a

spherical harmonic basis calculated using equation 10.

r i is the ith coefficient of the 16 coefficients that represents the reflectance function encoded

with the occlusion term in terms of a spherical harmonic basis calculated using equation 10 and

adhering to step 4.

6. When we have the solution to the integral, all we need to do is multiply this value by the colour of

the vertex to calculate the reflected light of the vertex. This is then summed with the emitted

radiance to achieve a solution to the rendering equation.

Notice that equation 11 misses out the inter-reflection term defined in equation 1. The only reason

for this is because the implementation provided does not include it. Inter-reflection can be encoded in

16

spherical harmonics just like the occlusion term V x , x ' was in this implementation. However, it is

more complex and therefore, not included here. Please refer to Green (2003) for a spherical harmonics

implementation that includes the inter-reflection term.

Comparing figure 8 and 9 shows what results can be achieved by introducing the occlusion term but

not including the inter-reflection term. Figure 8 shows the scene without the occlusion term and figure 9

shows the scene with the occlusion term. Both images are taken from Green (2003).

Figure 8. Figure 9. Using spherical harmonics, no occlusion term. Using spherical harmonics, with occlusion term.

Notice how the occlusion term improves the realism of the scene. The occlusion term models

whether or not any reaches a point and therefore, points “in the shade” appear darker than others; this has the

effect of introducing a correct notion of depth into the scene.

For completeness figure 10 shows the same scene with the inter-reflection term included. Notice how

the colours of adjacent areas in the scene seem to “leak” onto other areas, this is especially noticeable on the

white cup and on the floor by the red shape. This is due to light inter-reflecting around the scene before

heading towards the eye.

Figure 10 – Using spherical Harmonic with the inter-reflected term.

17

2.4.4 Weaknesses of Spherical Harmonic LightingAs already mentioned, calculating the result of the integral to the rendering equation (equation 1) for

a particular vertex is an off-line process. Therefore, there are grounds to suggest whether this technique is

truly real-time. The real-time element of this technique is that, once the lighting function and reflectance

function have been expressed in terms of spherical harmonic coefficients, the coefficients themselves can be

used in a real time scene.

In a real time scene spherical harmonics can be used efficiently to model static light in a static scene.

Rotating the light source or altering the scene in real time gives inaccurate lighting results because the result

of the integral to the rendering equation is stored per-vertex and is pre-computed off-line. It is possible to

rotate the lighting function by rotating the spherical harmonic coefficients for a vertex and perform equation

11 again in real-time, but this turns out to be inefficient and complex. Again, the reader is advised to read

Green (2003) to find out more.

Spherical harmonic lighting only models the diffuse term, and does not take into account the

specular term. This could be taken from the Phong reflection equation or calculated using a specular cube

map (see section 6.3.1 for a discussion on cube maps).

2.5 Summary

In this chapter we introduced the rendering equation and described how it accurately models

light/material interaction for non-translucent objects. We very briefly covered how the rendering equation is

solved in a non-real-time environment using a backward ray tracer and in section 2.3 and 2.4 we introduced

two techniques to model light/material interaction in real-time. Section 2.3 presented a simple technique that

is real-time friendly but lacks a certain degree of realism because it cannot account for occlusion and inter-

reflection. Section 2.4 presented a complex technique that can account for the occlusion and inter-reflection

terms but presents its own set of weaknesses.

18

Chapter 3

Research – Tools

3.1 Introduction

Fast rendering speed is paramount in this project because every scene has to render potentially

complex light and material interaction in real-time. Therefore, choosing the right tools is absolutely vital if

we want to gain the maximum execution speed from the programs. Also, an amount of flexibility in the way

we manipulate vertices and pixels is also needed.

This chapter will begin with a discussion on the rendering pipeline and manipulating it using

programmable and non programmable graphics hardware. Next, we will discuss how to program for

programmable graphics hardware using OpenGL and Direct3D as tools for compiling, linking and executing

programs that are compatible with modern graphics hardware.

Finally, this chapter will introduce Single Instruction Multiple Data (SIMD) as a way of optimising

3D computation and discuss how it can be embedded directly into a program using the Microsoft Visual

C++ compiler.

19

3.2 The Rendering Pipeline

The rendering pipeline is an abstract representation of the steps taken to project a 3D model made up

of vertices into a 2D array of pixels to be rasterized onto the computer monitor. A simplified diagram of the

rendering pipeline is shown in Figure 11.

Figure 11. Simplification of the

rendering pipeline.

3.3 The GPU and the Fixed and Programmable Rendering Pipelines

The Graphics Processing Unit (GPU) is analogous to the Central Processing Unit (CPU) that sits on

the motherboard of a PC. The GPU exists on the graphics card and has been heavily optimised for

performing vector and matrix calculations (Astle, 2006); something the CPU cannot offer. However, the

optimisation techniques implemented by graphics card vendors ultimately meant that there was very little

flexibility over how 3D models were rendered. For graphics programmers, this meant that rendering large

amount of vertices was guaranteed to be an heavily optimised operation but the little flexibility available

restricted the range of rendering algorithms available, e.g. Spherical Harmonic Lighting and Phong Shading

were not directly available to the graphics programmer using an API such as OpenGL or Direct3D. This

method of rendering is known as rendering using the Fixed Function Pipeline (Astle, 2006) (VanVerth and

Bishop, 2008). Here, the vertex attributes such as position, colour, texture coordinates and normal are

streamed from memory to the GPU through the rendering pipeline and the only control the programmer has

over the vertices once they enter the pipeline are through the render states of the graphics card. Such render

states include alpha blending (Duff, 1985) and Gouraud Shading (Gouraud, 1971). A link to a list of all the

20

Per Vertex Operations – At this stage individual

vertices are projected into screen space using the

world view and projection matrices. Shading

calculations such as Flat Shading and Gouraud Shading

(Gouraud, 1971) are performed at this step.

Clipping – Here, meshes that lie outside of screen

space are removed from the scene.

Vector Interpolation – Here vertex colours are

interpolated between other vertex colours in screen

space to calculate the initial colours of the pixels.

Per pixel operations – The final pixel colour is

determined here using alpha blending (Duff, 1985) etc.

Per Vertex Operations

In

Vector InterpolationVector Interpolation

Per Pixel Operations

Clipping

Out

Direct3D render states can be found at Microsoft (2008).

As GPU technology advanced programmers began to write microcode to be executed on the GPU

itself and hence alter the way vertices were handled once they were in the rendering pipeline. This

programmable pipeline allowed for a huge gain in control over how the graphics card manipulates data.

In OpenGL and Direct3D, the programmable pipeline allows the programmer to access and

manipulate the data in two areas of the pipeline, namely the steps responsible for per vertex operations and

per pixel operations. The next section goes into this in more detail.

3.4 Direct3D, OpenGL and Shaders

One particular area of background research for this project lies with finding out how to load, link and

compile programs to be executed on the GPU. There are several ways to do this but the most common

approach is to use a graphics API to access the graphics card of the computer. All of the literature that has

been read in preparation for this project uses one of two APIs. Astle (2006) uses the OpenGL graphics API

to prepare programs and send them to be executed on the graphics card. Whereas Luna (2006) uses a subset

of Microsoft's DirectX set of API's called Direct3D. A review of the literature of both Astle and Luna

suggests that both APIs allow the programmer to relay executable code to the graphics card's GPU in similar

ways albeit with a few differences. Astle and Luna describe two types of program that can be executed on

the GPU:

● The Vertex Shader – A program that executes once for each vertex, on each vertex sent to the

graphics card.

● The Pixel/Fragment Shader – A program the executes once for each pixel/fragment on each

pixel/fragment of the rendering surface (e.g. the computer monitor).

From now on this report shall refer to the two types of programs defined here as shaders.

Shaders are written by the programmer and compiled using routines made available by the graphics

API that are called by a rendering engine; a program that is responsible for loading and rendering graphics

and is built on top of OpenGL and/or Direct3D (see chapter 5). Compilation can be done at compile time of

the rendering engine, or at runtime of the rendering engine. Compiling the shaders at compile time of the

rendering engine leads to a faster execution of the rendering engine because the shaders do not have to be

compiled when the rendering engine executes. However, compilation of the shaders at engine runtime allows

the programmer to change the shaders after the rendering engine has been compiled; this is a good debugging

feature. Once compiled, the shaders are linked at runtime with the rendering engine and are streamed to the

graphics card followed by the vertex stream which is then manipulated using the shaders. The next section

will introduce a method for programming shaders in both OpenGL and Direct3D.

21

3.5 HLSL and GLSL

Both OpenGL and Direct3D provide their respective programming languages to allow the

programmer to program for the GPU. OpenGL's language is known as the OpenGL Shading Language or

GLSL and Direct3D has the High Level Shading Language or HLSL. Both languages have a C like syntax.

See the following examples.

//*****************************************************//Simple HLSL Vertex Shader written by Ben Wells (cuppatea83@hotmail.co.uk)//You can use it, just tell me. 8)//*****************************************************extern float4x4 g_matWorldViewProjection :

struct VS_INPUT{

float4 vertexPosition : POSITION0;};

struct VS_OUTPUT{

float4 tVertexPosition : POSITION0;};

VS_OUTPUT main(VS_INPUT input){

VS_OUTPUT output = (VS_OUTPUT)0;output.tVertexPosition = mul(input.vertexPosition, g_matWorldViewProjection);return output;

}

Figure 12. Simple vertex shader written in HLSL

Figure 12 shows how a vertex can be transformed from its local modelling space into clip space

using HLSL. Notice that a single global variable is declared, namely, extern float4x4

g_matWorldViewProjection. Here extern means that the programmer can set this variable from outside the

program itself and it is supplied directly, i.e. not as part of the vertex stream. Float4x4 is an HLSL data type

representing a 4 by 4 square matrix that will hold the product of the world, view and projection matrices.

Two structs are then declared; these are exactly like structs in C. Inside VS_INPUT a float4 type is

declared, meaning a 4 dimensional vector and it is marked with the input semantic :POSITION0. This input

semantic marks the register where the program can find the position data of the particular vertex that is being

operated on in the vertex shader; remember that this program executes on every vertex being passed down

through the pipeline.

Notice that there is a main function, as in C. The first line in this function creates a new instance of

VS_OUTPUT and sets the memory to null. Then the value located in :POSITION0 and represented with

VS_INPUT::vertexPosition is multiplied by the global variable g_matWorldViewProjection before being

returned back to :POSITION0 represented with VS_OUTPUT::tVertexPosition. From here the new value in

:POSITION0 can be picked up by the rest of the pipeline.

22

Here is the same code in GLSL:

//*****************************************************//Simple GLSL Vertex Shader written by Ben Wells (cuppatea83@hotmail.co.uk)//You can use it, just tell me. 8)//*****************************************************

void main(void)

{ gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;}

Figure 13. Simple vertex shader written in GLSL

The only differences here are that GLSL has some built-in terms that define the type of calculation

being performed, rather than specifically referencing registers with input semantics; as in the HLSL example.

The corresponing fragment/pixel shader will calculate the colour of every projected vertex as well as

the colour of every interpolated screen space position between the vertices. The colour can be hard coded or

can be sampled from a texture. Figure 14 shows a HLSL pixel shader that sets the vertex colour to white.

//*****************************************************//Simple HLSL Pixel Shader written by Ben Wells (cuppatea83@hotmail.co.uk)//You can use it, just tell me. 8)//*****************************************************

float4 main() : COLOR0{

return float4(1.0f, 1.0f, 1.0f, 1.0f);}

Figure 14. Simple pixel shader written in HLSL

Both GLSL and HLSL support a multitude of operations similar to the C language and both provide

support for texture coordinate look up and setting render states.

3.6 Direct3D Effects Framework

Unlike OpenGL, Direct3D provides a nice method for writing both fragment/pixel shaders and

vertex shaders. This method is called the Effects Framework whereby both vertex shaders and pixel shaders

are written in a single file and are grouped with logical structures called Techniques. Each Technique can

have one or more Passes where each Pass contains a vertex and/or pixel shader. This allows the programmer

to logically organise very complex shading algorithms into multiple Passes and each version of a complex

algorithm can be executed with the correct Technique depending on the graphics hardware of the end user. A

simple HLSL Effect is given in figure 15.

23

//*****************************************************//Basic HLSL Effect written by Ben Wells (cuppatea83@hotmail.co.uk)//You can use it, just tell me. 8)//*****************************************************uniform float4x4 g_matWorldViewProjection :

struct VS_INPUT{

float4 vertexPosition : POSITION0;};

struct VS_OUTPUT{

float4 tVertexPosition: POSITION0;};

VS_OUTPUT vs_ main(VS_INPUT input){

VS_OUTPUT output = (VS_OUTPUT)0;output.tVertexPosition = mul(input.vertexPosition, g_matWorldViewProjection);return output;

}

float4 ps_main() : COLOR0{

return float4(1.0f, 1.0f, 1.0f, 1.0f);}

technique Basic{

pass Pass_0{

VertexShader = compile vs_1_0 vs_main();PixelShader = compile ps_2_0 ps_main();

}

}

Figure 15. A simple HLSL Effect that groups a vertex and pixel shader into a single file using a single Pass under a single Technique.

Programmers can write Effects in any text editor and compile them using the routine

D3DXCreateEffect provided as part of the Direct3D API to create an ID3DXEffect object which can be used

to set externs declared in the effect file. To send the compiled Effect file to the graphics card the programmer

should call ID3DXEffect::SetTechnique followed by ID3DXEffect::Begin and then

ID3DXEffect::BeginPass. Rendering resumes as normal after these steps. Figure 16 shows the function

prototype for D3DXCreateEffect in C++. See Microsoft (2008b) for more information.

HRESULT D3DXCreateEffect( LPDIRECT3DDEVICE9 pDevice, LPCVOID pSrcData, UINT SrcDataLen, CONST D3DXMACRO * pDefines, LPD3DXINCLUDE pInclude, DWORD Flags, LPD3DXEFFECTPOOL pPool, LPD3DXEFFECT * ppEffect, LPD3DXBUFFER * ppCompilationErrors);

Figure 16. Compiling an HLSL effect using Direct3D in C++.

24

3.7 SIMD

As mentioned in section 3.3, the GPU on a graphics card has been heavily optimised for vector and

matrix calculations. However, some operations that only need to be executed once per frame cannot and

should not be executed on the GPU. For example, each model will have a matrix that combines the product

of the world, view and projection matrices for that model. It would be a waste of GPU cycles to send the

separate world, view and projection matrices to be multiplied within a vertex shader because this

multiplication would be performed once per vertex and we know that the answer to the multiplication is the

same for every vertex; i.e. it only needs to be calculated once. So the calculation should be done using the

CPU. The question is how can we optimise this multiplication to rival the 3D calculation speeds of a GPU?

The answer lies in Single Instruction Multiple Data (SIMD) programming.

SIMD is a form of assembly program that turns out to be particularly suitable for vector calculations.

Both Intel and AMD offer their own SIMD support on most modern Intel and AMD CPUs. The CPU used in

this project is an Intel CPU therefore the research undertaken applies to Intel only. Zerbst and Duvel (2004)

outline Intel's Streaming SIMD Extensions (SSE) as a way to perform fast vector calculations on the CPU

using sets of four 32 bit floating point numbers. Ever since the launch of the Intel Pentium III CPU, SSE

programmers have been able to make use of eight 128 bit registers on the CPU, these registers are labelled

XMM0 – XMM7 and are particularly suited to storing four 32 bit floating point numbers each. Notice that

3D computer graphics often represents 3D vectors in a homogeneous coordinate system with x, y, z, and w

values. It should be clear to the reader that a single SSE register can hold a single vector defined in a

homogeneous coordinate system.

The benefit of using SIMD and SSE is that two 128 bit registers can be operated on in a single cycle

of the CPU. This has the potential to greatly reduce the number of CPU cycles used in vector calculations

and, therefore, speed up rendering. Furthermore, SSE instructions can be directly embedded in C++

programs using the Microsoft Visual C++ compiler. Figure 17 and 18 compare the steps taken to multiply

two vectors defined in a homogeneous coordinate system. Notice the figure 17 requires four separate

multiplies and figure 18 shows that using SIMD reduces the number of multiplies to one.

25

Vec4 mul(const Vec4 & v1, const Vec4 & v2){

D3DXVECTOR4 vOut;vOut.x = v1.x * v2.x;vOut.y = v1.y * v2.y;vOut.z = v1.z * v2.z;vOut.w = v1.w * v2.w;return vOut;

}

Vec4 mulSIMD(const Vec4 & v1, const Vec4 & v2){

Vec4 vOut;__asm{

mov edi, v1mov esi, v2movups xmm0, [edi]movups xmm1, [esi]

//This line multiplies v1 and v2 in one CPU cyclemulps xmm0, xmm1

//----------------------------------------------------------movups [vOut], xmm0

}return vOut;

}

Figure 17. Standard C++ vector multiplication. Figure 18. SIMD vector multiplication.

3.8 Summary

This chapter discussed the research that was done into the tools for this project. The graphics

pipeline was discussed as well as how programmable graphics hardware can increase the flexibility of the

pipeline using languages such as HLSL and GLSL to program vertex and pixel shaders that execute on the

GPU. The Direct3D Effect Framework was shown to be an elegant way to write programs for programmable

graphics hardware and SIMD programming was introduced as a way to optimise vector calculations that

should be done on the CPU rather than the GPU.

This marks the end of the research element to this report. The next chapters will discuss

methodology, design and implementation of the rendering engine and the scenes that make up the practical

part of this project.

26

Chapter 4

Design & Implementation – Methodology

4.1 Overview of Methodologies

It is clear that traditional business software development methodologies such as the Waterfall model

(Royce, 1970) or the V model (Bucanac, 1999) are not suitable for this project. This is due to the research

nature of the project as opposed to the business application nature suggested by the Waterfall and V models.

However, the project can take some aspects of the Waterfall and V models into account. For instance,

the specific stages of requirements, design, implementation and verification of the Waterfall model still

apply here but in a different sense:

● Requirements - In this project requirements fall in closely with feasibility; it is required that the

project should model situations that have sufficient background information on them, for instance.

● Design- This is comparable with the particular mathematical models that the project will use to build

27

realistic graphical scenes.

● Implementation- This is essentially the same in the project as what is proposed by the Waterfall

model.

● Verification – This represents the public's assessment of the graphical scenes, they are essentially

verifying the scenes' realism.

However, the structure of the Waterfall model does not lend itself to this particular project. This is

because the project requires an implementation of multiple scenarios that model the real world and this

notion lends itself to more of an iterative development process.

Such iterative methodologies do exist, in particular the Spiral model (Boehm, 1988) lends itself to

iterating over system prototypes. However, whilst this project can take into account the iterative nature of the

Spiral model, its aim is not to develop prototypes as such. This is because the iterations of the project are not

over a single application, but over multiple models that each require different code and methods.

4.2 Custom Methodology

This project combines an builds on the methodologies mentioned above into a custom methodology

that suits the needs of the project and helps guide it towards delivering good quality research. The custom

methodology is loosely iterative, i.e. not reworking the same code but iterating over the same development

process in order to add to the overall application after the end of each iteration.

After much deliberation the custom methodology consists of the following stages:

1. Do the ground work (build the system to load, compile, link and execute programs on the GPU).

2. Decide on a scenario that is to be modelled (flowing water, glass, etc).

3. Evaluate the model's feasibility (read up on the physics, and see if it has been used in computer

graphics. Decide how easy the model would be to implement on the GPU).

4. Feasible? No = go to stage 2. Yes = go to stage 5.

5. Implement a scenario using GPU programming technologies. Try and make it as realistic as possible

by comparing the visuals to real-life films and photographs and test the scene's rendering speed

using timers built into the the rendering engine (see chapter 5).

6. Add the scene to the “bank of scenes”.

7. Is there time to implement one more? Yes = go to step 2, no = go to step 8.

8. Let the public evaluate the scenes.

9. Process the results.

28

10. Conclude and propose the computer graphics equivalent to the Turing Test.

This methodology is heavily customised to this particular project but can be re-customised to other

research projects that sum the results of each iteration in to a “bank” to be evaluated at a later stage.

29

Chapter 5

Design & Implementation – The Rendering Engine

5.1 Introduction

Chapter 4 of this report mapped out the custom methodology for the project. Here the report will

discuss the techniques used that address parts one of the custom methodology. In particular , the design

decisions made to build a rendering engine that will take shaders written with a text editor and load, compile,

link and execute them efficiently. The rendering engine also needs to handle vertex and texture information

efficiently; this will also be discussed.

5.2 Design Decisions and Justifications

5.2.1 Graphics APIAs mentioned in section 3.3, one way to execute programs on the GPU is to use a graphics API such

30

as OpenGL or Direct3D as a tool for loading, linking, compiling and executing shaders. It is sensible to

suggest that the wheel should not be reinvented and this project should make use of the tools provided by one

of these APIs. Both OpenGL and Direct3D were considered and it really did not matter which API was

chosen because both can satisfy the requirements for the project.

Ultimately, Direct3D was chosen because of the following reasons:

● Direct3D is the industry standard for programming rendering engines on Microsoft platforms such as

the XBox 360, and Microsoft Windows.

● My career after university will use Direct3D, therefore, gaining experience with it as part of a large

project is advantageous.

● I already had a little experience programming with Direct3D.

● Direct3D offers a more elegant and more intuitive approach to using shaders than OpenGL, this is

due to the Effects Framework.

5.2.2 Programming Language

Direct3D is written to be fast executing therefore it is written in C++ and assembler. This approach

also gives the programmer access to the computer at a very low level, enabling heavy optimisation using

techniques such as SIMD (section 3.7) amongst others. One of the requirements for this project was that the

graphics should run in real-time therefore it makes sense for the project to follow the same logic as the

Direct3D programmers and use C++ and assembler. Familiarity with C++ is another factor; I am very

familiar with C++ and it makes sense not to struggle with a programming language.

The shaders would be written in HLSL because this programming language is compatible with

Direct3D and is easy to learn due to the C like syntax.

5.3 Manager Modules

A rendering engine is a fairly complex piece of software and the sharing of different resources

between different parts of the system needed to be handled in an elegant way to control the scalability of the

system. To help manage all of the resources, manager modules were introduced. The manager modules are

different components in the system that manage resources such as textures and shaders. Manager modules

also hold pointers to other manager modules, therefore, enabling communication between them. The three

main manager modules are outlined below:

● The top level manager module was the Scene Manager. This module was used to hold handles to the

other managers in the system. It also had the task of creating and holding a constant handle to the

graphics card on the PC and was used as an interface into the graphics card for the main rendering

system.

31

● The system also had an Effects Manager. This was used to compile and store all of the compiled

shader code and to set external variables within the shaders themselves. The Effects Manager was

also used to stream the shaders to the graphics card via the Scene Manager.

● The system had a Texture Manager. This was used to stream texture data from hard disk into RAM

and finally to the memory on the graphics card. This manager also performed some basic memory

management by making sure textures were only loaded into memory when needed.

The system also contained other managers to handle timers, memory, loading 3D models and

managing Direct3D itself. The system used SIMD where possible to optimise CPU vector and matrix

calculations.

5.4 Summary

This chapter discussed the design decisions made when implementing the rendering engine for this

project. It began by discussing the graphics API used for this project before moving onto the programming

language. The final part of this chapter outlined the partitioning of the rendering engine into manager

modules and gave an overview of the main ones.

32

Chapter 6

Design & Implementation – The Shaders and Techniques

6.1 Introduction

The previous section discussed some of the approaches that were taken in this project to build the

rendering engine that would load, compile, link, and execute shaders. This engine can also load models and

send the vertex data to the graphics card and have the shaders execute on these models. This section will

discuss the shaders themselves, and describe the techniques used to generate the effects seen in the final

videos. This section will link in some of the background research on light/material interaction and address

translucent objects; something not factored into the rendering equation.

One of the goals of this chapter is to discuss, in detail, the maths of the implemented scenes and how

the maths was represented inside a shader. The report will address the water scenes, the glass scene and the

scenes that modelled the sub-surface scattering of light.

Another goal of this chapter is to show the results of applying steps 2 – 5 in the custom methodology

proposed in section 4.2.

33

6.2 The Water Scenes

Several scenes modelled as part of this project model water and there were two main elements of

water that were taken into consideration, namely water movement and how light reacts at the barrier that air

makes with water. Section 6.2.1 aims to demonstrate how water movement was modelled and section 6.2.2

aims to demonstrate how light/water interaction was modelled.

6.2.1 Geometric Water MovementThere are several ways of modelling the movement of water. Such methods include the Fast Fourier

Transform approach adopted by Tessendorf (2004), the summing of sine waves approach outlined by Finch

(2004) and Gerstner Waves, also outlined by Finch (2004). Here the report outlines the summing of sine

waves approach because this is what has been adopted as part of the water scenes for this project.

A 2D plane stretching out over a 3D environment can be represented using a Cartesian coordinate

system where the plane stretches out over the X and Y axes. The height of point (x,y) is specified at a

particular time t using the function H(x,y,t). In order to get a realistic looking water surface, the height

function must be chosen carefully as to best represent the surface of real water. The sum of sine waves

approach uses a combination of sine waves with varying parameters to give the realistic look of a water

surface. This function, as described by Finch (2004) , is shown as follows:

H x , y ,t =∑i

W ix , y , t , (12)

where,

W ix , y ,t =2 Ai sin Di⋅x , y wii t 12

k

(13)

where,

Ai is the wave amplitude for the ith wave,

Di is the direction the wave is travelling over the 2D plane for the ith wave,

w i is the frequency of the wave given as 2

wavelengthi for the ith wave,

k is the steepness of the wave,

i is the phase constant for the ith wave and is responsible for the overall speed of the rolling motion

of the wave and is expressed as 2waveSpeed i

wavelengthi,

t is time and is increased every frame by the rendering engine.

34

What the reader should notice about this model is that the height of each vertex of the water mesh is

generated by summing together an arbitrary amount of sine waves that are each shifted across the surface of

the mesh according to t . Notice also that the initial position of the height is determined by the x , y location of the vertex and, because these values linearly increase, a smooth sine wave is generated across the

mesh.

It is possible to represent this model in a vertex shader because summations translate directly into for

loops in a programming language; the pseudocode is shown in figure 19 where getHeight() is a function

which evaluates equation 13 (remember that the vertex shader executes on every vertex in the model).

//*****************************************************//Z is global up//*****************************************************

vertexPosition = getVertexPosition();

for(i = 0 to NUM_WAVES)

//getHeight(X_POS, Y_POS, WAVE_NUMBER, TIME) vertexPosition.z += getHeight(vertexPosition.x, vertexPosition.y, i, TIME_T); end

Figure 19. Pseudocode for calculating the vertex height.

6.2.2 Light/Water Interaction

The way light reacts with water is very complex and there are several phenomena that occur. For

example, when viewed from above the water surface, areas of the ocean floor appear to distort with water

movement as light is refracted as it exits the water into the air. Another thing to note is that some light is

totally reflected off the water surface. Also, the water caustics described by Guardado (2004) show how light

is reflected and refracted as it passes through the water surface to form patterns of light on the ocean floor.

It was decided that water effects such as these should be modelled to improve the realism of the

water scenes. In particular the distortion of the ocean floor as seen from an observer above the water.

However, upon further investigation, it was deemed not feasible to model the patterns on the ocean floor as

these require a lot of computational power compared to the gain in realism of the scene.

As with many light/material interactions such as Phong shading or spherical harmonic lighting, the

vertex normal needs to be present in order to perform the calculations but because our model of water

movement changes the height of every vertex every frame, the normal will also change every frame and,

therefore, needs to be calculated every frame. The next section aims to show how this was done.

35

6.2.2.1 Calculating the Normal of Each Vertex

Due to the continuous nature of the height function it becomes possible to calculate the normal for

every vertex on the plane. It is essential to able to calculate the normal because it is needed for light

refraction and reflection calculations.

To calculate the normal for each vertex we first need to calculate two vectors that are tangent to the

vertex. We do this because the cross product of these vectors will yield the normal to the vertex.

A vertex on the plane can be represented by the following:

P x , y ,t = x , y , H x , y , t (14)

The first tangent vector, namely the bi-tangent, is represented as B(x,y,t). This vector can be

calculated by finding the partial derivative of the vertex's position with respect to the x coordinate. The

second tangent is represented as T(x,y,t) and can be found by finding the partial derivative of the vertex's

position with respect to the y coordinate. In other words we calculate the gradient with respect to x and then

y. Hence, we obtain:

B x , y , t = 1,0, ∂∂ x

H x , y , t (15)

T x , y , t = 0,1, ∂

∂ yH x , y , t (16)

Given the two tangent vectors found using equation 15 and 16 the normal can be found by applying the cross

product:

N x , y , t = B x , y , t × T x , y , t (17)

Hence, we obtain the vector for the normal:

N x , y , t = − ∂∂ x

H x , y , t ,− ∂∂ y

H x , y ,t ,1 (18)

where,

∂∂ x

H x , y , t = ∑i

∂∂ x

W ix , y ,t (19)

where,

∂∂ x

W ix , y , t = Di . xk wi Ai cos Di⋅x , y w ii t sin Di⋅x , y w ii t 12

k−1

(20)

Note that the partial derivative with respect to y is not shown; it follows the same procedure, however.

36

Now that the vertex normals have been calculated, we can apply the shading models outlined in

chapter 2 if we wish. The water videos in this project used the diffuse and specular terms given by Phong

reflection (section 2.2.4). The specular intensity was given as the colour white and the diffuse intensity was

calculated using a texture look up where the texture coordinates were calculated using the method described

in the next section of this report.

6.2.2.2 Modelling Refraction

When moving water is viewed from above, the shapes of objects underneath the water appear to

distort as waves pass over the top of them. This is due to a phenomenon known as refraction. To model this

mathematically, each media (water and air) is given a refractive index that, when grouped with the refractive

index of the other medium forms a ratio that is equal to the ratio of the sines of the angles formed between

the normal to the boundary and the incoming and outgoing rays of light. This is known as Snell's law and is

given in equation 21 and shown in figure 20.

sin1

sin 2=

n2

n1 (21)

Here n1 and n2 are the refractive indices of the materials that the light is exiting and entering

respectively. A refractive index of a material is a measure of how much the speed of light (or any other wave)

is reduced inside the material. For example, the refractive index of water is 1.333 and the refractive index of

a vacuum is 1.0, therefore light propagates through water at about 0.75 times the speed it does in a vacuum

(1/1.333).

Figure 20. Snell's Law.

Calculating the refraction vector, T, given in figure 20 is non trivial and very computationally

inefficient because the calculation needs to be done on every pixel in a pixel shader to give the best results

(we are now determining the final colour of a pixel). This combined with the computational load already

37

used to calculate the vertex movement in the vertex shader would have made the scene run at unacceptable

frame rates.

The decision was made to implement a simplification of refraction that, whilst not physically

accurate, does give good looking results. This simplification reflects vector E given in figure 20 about the

vertex normal and then projects a scaled down version of this reflected vector onto the ground plane. This

new vector is then used as a refraction term that alters the texture coordinates of a pixel. These new texture

coordinates are then used to look into a texture to retrieve the final pixel colour. Equation 22 (Foley et al.

1996 ) shows how to calculate the true refraction vector T and equation 23 and 24 shows how I calculate the

final texture coordinate P of a pixel on the water surface.

T=N n1

n2E⋅N 1 n1

n2 2

E⋅N 2−1n1

n2E (22)

Below is how this project distorts the texture coordinates of the pixel.

P = O−M xy (23)

where,

M = k E−2 N E⋅N (24)

k is a scalar that controls the overall distortion of the texture coordinates and O is the original texture

coordinates of the pixel.

Notice how the refraction simplification shown in equations 23 and 24 is much more

computationally efficient than the true refracted vector given in equation 22. Using the simplification is not

physically accurate but the saving in computational power makes it worth the sacrifice and therefore

equations 23 and 24 were applied in the pixel shader of the water scenes to model refraction. The intuitive

thinking behind this simplification is that as 1 grows in equation 21, 2 must also grow to hold the

ratio of the refractive indices. This has the effect of pushing the refracted ray along the ocean floor. The same

can be said about projecting the reflected ray onto the ocean floor, as 1 in equation 21 grows the

projected reflection ray grows along the ocean floor. The scaling factor k is then added to compensate for

the refraction effect; as k tends towards 0, the water becomes less and less refractive. This means that

k must be carefully chosen to give the true impression of refraction.

6.3 The Glass Mummy

In the last section we modelled light and its interaction with water. We also modelled the movement

of water by displacing the vertex positions of the water mesh in a vertex shader. However, due to the

computational load of displacing the vertex positions we could not apply Snell's law to model refraction.

38

This scene aims to allow us to model refraction properly by taking away the need to displace the vertex

positions and using the saving in computational power to apply proper refraction. This scene also aims to

look at reflection in more detail and apply physically real reflections to the glass model by modelling total

internal reflection. Finally we model the blending of the refracted and reflected components into a single

model that creates the look of glass.

6.3.1 Using Snell's Law To Model RefractionRecall Snell's law given in equation 21 and figure 20. This section aims to show the reader how to

use Snell's law to render refraction of light through a glass object. Using equation 22 we can the calculate the

refracted vector, T, and using this vector we can index the colour of the pixel using a cube map; also known

as a 3D texture. A Cube map can be thought of as a cube sitting in 3D spaces with edges of length 1. Each

face of the cube represents a texture and a pixel on that texture can be indexed using a 3D vector (as opposed

to a 2D vector with ordinary texturing) with origin at the centre of the cube; see figure 21. This approach is

extremely useful for looking up the pixel colour taking refraction into account because we can use the

refracted ray given by equation 22 as an index into the 3D texture directly.

Hopefully the reader can see that we can determine the refracted component of the pixel colour by

using the refracted vector, T, given by equation 22 as the lookup vector into the cube map.

6.3.2 Modelling ReflectionWe now have the refraction colour of the pixel, next we need to model light reflecting off the glass.

The first step to modelling this effect is to treat the glass as if it were perfectly reflective, i.e. a mirror. To do

39

Figure 21. Cube Mapping – If we want to index the colour of a pixel on the texture given on the left then we cast a 3D ray from the centre of the cube on the right to intersect with the pixel as if the texture were wrapped round a cube. The various dots on the image on the left match to the corresponding dots on the cube on the right.

this, cube maps were used again. However, this time the reflected ray was calculated using equation 25 and

this ray was used as an index into the cube map to retrieve the reflected pixel colour.

R = E−2N E⋅N (25)

Here, E and N are equivalent to those in figure 20.

6.3.3 Combining Refraction and ReflectionSections 6.3.1 and 6.3.2 described how to calculate the refracted and reflected components of the

pixel colour by using the reflected and refracted rays to index a cube map to produce two colours per pixel;

one colour representing the reflected light and one colour representing the refracted light. To achieve the

final effect both of these colours need to be combined. In order to combine them in a physically realistic way

we need to calculate the amount of light reflected and refracted at that point on the surface of the glass. The

reflected component equates to the part of the incoming light that is totally internally reflected and the

refracted component can be thought of as part of the incoming light that has passed through the glass itself.

To calculate the correct mix of reflected and refracted light we need to find the value, p, given in

equation 26. Astell (2006) shows that Fresnel's equations can be used to find this value. One form of

Fresnel's equations are shown in equations 27 and 28 where each equation represents one of the two

polarised orientations for the reflected light from the point on the object using the refraction indices, n1

and n2 of the light transition media. In this project the light transition media were glass and air with

refractive indices of 1.5200 and 1.0003 respectively.

FinalColour= p reflectedColour 1− prefractedColour (26)

R s= n2 cos 2−n1cos 1n2 cos 2n1cos 1

2

(27)

R p= n1 cos2−n2cos 1n1 cos2n2cos 1

2

(28)

where 1 and 2 are the same as those given in figure 20.

As already stated, equations 27 and 28 calculate the two polarised orientations for the reflected light.

As human eyes cannot see the polarisation of light we factor out the polarisation by averaging R p and

R s to get the final value for p shown in equation 29.

p=R sR p

2 (29)

40

One weakness of this method is that it does not model the refraction of light when it exits the glass.

This can be taken into account by using a ray tracing method but one questions whether this may lead to the

scene being to complex too render in real time.

The results of applying these techniques can be scene in the glass mummy video prepared as part of

this project.

6.4 The Wax Models

The aim of the wax model scenes was to model a property of light/material interaction that the

rendering equation does not capture; the sub-surface scattering of light. This is when light enters a material

and is inter-reflected within the material itself before eventually being reflected back out to the surrounding

environment. Many real world materials such as wax, skin, milk and marble exhibit this property and it has

been heavily researched because including sub-surface scattering into your rendering engine and scenes can

drastically improve the realism of a material. The Nvidia Human Head shown in figure 1 uses a model of the

subsurface scattering of light to greatly improve the realism of the skin.

This project modelled sub-surface scattering in a very simplistic way to maximise rendering speed.

The assumption that was made for this model was to assume that when light enters an object it only

attenuates as it makes its way through the medium. This attenuation is due to the light being absorbed by the

medium itself. If the light beam becomes fully absorbed it will not exit the medium. If the light beam is

partially absorbed by the medium then a less intense beam of light will result as it exits the medium.

This assumption makes the physics of this model of subsurface scattering very simple, much simpler

than models proposed by Jensen et al (2007). But simplicity leads to fast rendering speeds and therefore this

method renders in real-time. Equation 30 shows the function used to calculate the intensity of light that exits

a medium at a given pixel given the thickness of the medium and the intensity of light that entered the

medium.

I o=I ie−kt (30)

where,

I i is the intensity of light that enters the medium.

t is the length of the vector between the entry point and exit point of the light ray within the medium, also

seen as the thickness factor.

k is the absorption constant.

Theoretically, equation 30 is very simple. However, incorporating it into computer graphics is an

involved process that requires the scene to be rendered twice for every frame; once to record where the light

ray enters the medium and once to record where the light ray exits. The algorithm is as follows:

41

1. Render the object as scene from the position of the light source.

2. For every pixel on the object, record the distance between the near viewing plane and the pixel in a

texture; d 1 .

3. Render the scene from the normal viewing position.

4. For every pixel position, P , on the object, project that pixel into the screen space of the light

source to get P l then calculate a set of 2D texture coordinates, P ' ' , using equations 31 and 32.

This enables us to find the correct d 1 from the texture of distances created in step 2.

5. Calculate the distance between the camera and P and store as d 2 .

6. Calculate the distance between d 1 and d 2 ; this is the value t in equation 30.

7. Calculated the intensity of p using equation 30 to get I o ; this is the colour of the pixel at pixel

position P .

P ' =P xy

2 Pw 0.5,0.5 (31)

P ' ' = p ' x , 1−p ' y (32)

where,

P xy are the x and y coordinates of screen space pixel position P l .

Pw is the w coordinate of screen space pixel position P l .

P ' x is the x coordinate of P ' .

P ' y is the y coordinate of P ' .

As already said, the physics of this method is fairly straight forward, but representing the physics in

computer graphics is not. This is the only scene in the project where representing the scene as computer

graphics was less straight forward than the physics that back it up. However, using this model gives good

looking results and renders in real-time at 60 frames per second even though the scene needs to be rendered

twice.

42

6.5 Summary

In this chapter the aim was to describe how sections 2 – 5 were applied to the practical side of the

project.

The chapter discussed modelling water movement by summations of sine waves and how to

calculate the normals to these waves every frame in order to implement light reflection and refraction. We

discussed the feasibility of using a physically realistic model of refraction for the water and examined a new

approach to estimate refraction by distorting texture coordinates.

We also discussed the modelling of glass and the use of cube maps to calculate the pixel colour using

the true refraction vector as well as reflection colour using a reflection vector. We then discussed how

Fresnel's equations were applied to calculate how much light arriving at the eye is reflected light and how

much is refracted light.

We finally discussed a simplified model for rendering the effect of sub-surface scattering, a

phenomenon of light/material interaction that the rendering equation does not capture. We discussed an

algorithm that required the scene to be rendered twice but gave good looking results despite the simplicity of

the physics.

The next chapter of the report will discuss the evaluation of the scenes and will aim to address sections 8

and 9 of the custom methodology.

43

Chapter 7

Evaluation

7.1 Introduction

This chapter of the report aims to discuss how steps 8 and 9 in the custom methodology were

addressed in this project and to evaluate whether the project objectives stated in chapter 1 of this report were

met.

The first section of this chapter will go through the objectives of this project and discuss how close

the project was to satisfying the objectives.

The second section of this chapter will discuss the method used to evaluate the scenes produced in

the project and how this method applied to sections 8 and 9 of the custom methodology.

The third section of this chapter will outline the results of the public evaluation and briefly discuss

some of the relevant comments made by the public.

The fourth section of this chapter will discuss how this project could be improved if the project were

to be tackled again. This section will also address how the project could be extended if more time was given.

The main theme of this chapter is to see how the realism of the scenes stood up to the scrutiny of the

public and how their comments were used to evaluate whether the objectives were met.

44

7.2 Evaluating the Objectives

The first objective stated in chapter 1 was to gain fluency with the concepts of programming

graphics hardware. This was the foundation of this project because the mathematical models used to generate

the scenes had to be translated into computer graphics using programmable graphics hardware. The work

undertaken as part of the tools research for this project as well as practice implementations of Phong shading

and spherical harmonic lighting aided in this. As did an understanding of C and C++ whose syntax is close to

HLSL; the programming language used to generate the shaders for this project. The final proof that this

objective was met is that eight scenes were generated that translated several mathematical models into

computer graphics; far more than the one scene required as part of my minimum requirements.

The second objective states that the project should involve the study of an API specific

programming language that can be compiled and executed on graphics hardware. As already mentioned.

eight scenes were developed using HLSL and further research into the Direct3D Effects Framework enabled

the shaders used in this project to be written elegantly and efficiently.

The third objective of this project was to build a flexible and efficient rendering engine. All of the

scenes produced as part of this project execute in real-time. Research into SIMD and simplifications in

mathematical models produced an engine that was able to render fairly complex mathematical models of

light/material interaction and vertex movement efficiently.

The fourth objective states that research must be done regarding the mathematical models of light

and its interaction with materials. This has clearly been met.

The fifth objective stated that graphical scenes should be produced displaying visual complexity

combined with efficiency. Nowhere is this more apparent than the refraction effect in the water scenes, where

the visual appearance of refraction was produced by compromising on physical realism in exchange for a

more efficient implementation.

The sixth, seventh, eighth and ninth objectives are met as part of the evaluation of this project and

are discussed in the remainder of this report.

7.3 Public Evaluation Method

Sections 8 and 9 in the custom methodology state that the public should evaluate the scenes

produced for this project. It has also been stated that the results gained by the public evaluation should be

processed in some way as so to get meaning from their comments. This is so that their comments can be used

to address the evaluation of the objectives for this project.

The platform used for the public evaluation was the WWW. A website was written that contained

screen captured recordings of the scenes produced. Users are able to go directly to

45

http://www.benwells3d.com/ and begin a series of steps where they are able to watch and comment on the

scenes produced. The comments come in the form of ratings; i.e. 1 to 10, and written comments.

The series of questions needed to address how realistic the scenes are, and what parts of the scenes

succeeded and failed in generating the illusion of realism. The question of how to increase the realism was

also needed. Due to these reasons, each scene posed the following questions:

1. How realistic is the video? This was answered on a scale from 1 to 10. It was hoped that a quantified

value for the realism of the scenes will be produced by getting the public to address the realism of

the videos in this way.

2. Does the scene accurately reflect the description? Again, this was answered on a scale from 1 to 10.

It was hoped that this question would quantify how close the scene is to what it is trying to represent;

i.e. How close does the scene match its real life equivalent. For example, does this glass look like the

right kind of glass? Or, does it look like clear plastic? Or, perhaps it doesn't look at all like its real

life equivalent.

3. What do you like most about the scene? This question required a written answer and it was hoped

that by posing this question the users would pick up on the parts of the scenes that were most

realistic and, therefore, which mathematical models translate particularly well into computer

graphics.

4. What do you like least about the scene? This question required a written answer and it was hoped

that by posing this question the users would pick up on the parts of the scenes that did not quite

work. This could then be translated into which mathematical models did not translate well into

computer graphics.

5. What (if anything) would make the scene appear more realistic? This was the final question and it

was hoped that this would address the need to improve the scenes and provide evidence on how the

project as a whole could be improved. It was also hoped that the user may suggest additional features

for a particular scene that were not modelled in this project. These features could then be

implemented if the project were to be extended.

Once the website for public evaluation had been provided and was uploaded to the internet several

steps where carried out in order to let the public know about the website. Ken Brodlie (University of Leeds)

very kindly let me give a fifty minute presentation to the second year undergraduate Computing students at

the University of Leeds. The subject of this presentation was to introduce the students to programming

shaders with Direct3D. The presentation rounded off by introducing this project and providing the URL for

the evaluation website. Ken Brodlie also told his first years about the evaluation website and I introduced

family and friends to it. Overall, 20 people took part in the evaluation.

46

7.4 Evaluation Results and Improvements

In the last section we outlined the evaluation method for the videos produced as part of this project.

In this section we will discuss some of the strengths, weaknesses and improvements of the scenes with

regards to the initial project aim outlined in chapter one.

Of the eight videos produced, table 1 shows how realistic they appeared to the general public when

compared to their real life equivalents. This value for realism was taken as an average to the answers given

to question 1 posed on the website and match description was taken as an average to the answers given to

question 2.

Genre Video Url (YouTube) Realism (%)

Match Description (%)

Sub-surface scattering http://www.youtube.com/watch?v=entEiYsSgq4 74.3 85.3

Water http://www.youtube.com/watch?v=Mc6MNISFeTs 86.9 85.3

Water http://www.youtube.com/watch?v=Z9iqPbaQp88 87.5 85.0

Sub-surface scattering http://www.youtube.com/watch?v=aeio0nUqycY 91.8 96.4

Water http://www.youtube.com/watch?v=k8LahgUNQPs 77.0 90.0

Water http://www.youtube.com/watch?v=U1KL3JgwUiw 90.0 100.0

Sub-surface scattering http://www.youtube.com/watch?v=0qlf-nl8r3g 88.9 98.0

Glass http://www.youtube.com/watch?v=5F3e9FOODIQ 95.5 97.2

Table 1. Realism results given by the public evaluation.

The results given in table 1 are fairly promising with respect to the initial project aim which stated

that this project should build realistic models of light and its interaction with materials using computer

graphics.

The glass video in particular scored very highly for both realism and description. The description for

this video stated that the model was thick “Coca Cola” bottle glass and therefore the public were comparing

my model to that kind of glass. One user stated that the lighting effects modelled in this glass video were

very believable. This comment can be taken as evidence that the reflection and refraction models used to

create the look of glass created a believable look for the scene. One user stated that he/she was not sure

where the light source was coming from. This comment was justified because the light source was captured

within the cube map and therefore not modelled using point light sources. The user then suggests adding

light sources to improve the specular effect. This possible improvement in the scene could be achieved using

a technique such as Phong shading with the specular term of the Phong reflection model. It is questionable

whether this would add or take away from the realism of the scene, however. With more time this could be

47

implemented as part of a further solution and evaluated with another public evaluation.

The video that was rated the least real was the first sub-surface scattering video that received a

realism score of 74.3%. The most common complaint about this video was the lack of detail in the mesh.

This leads me to suggest that the sub-surface scattering technique used in this project does not work well

with low detailed meshes. This is a feasible assumption because the amount of detail seen in a mesh using

this effect for sub-surface scattering requires there to be a large variation of the thickness parameter (defined

in equation 30) across the mesh. A more physically realistic mathematical model for modelling sub-surface

scattering may give a more realistic appearance to lower detailed meshes.

One of the main comments by the public regarding the water scenes was that the direction of flow of

water was too uniform and that patterns formed by the waves were not random enough. One user commented

that to improve the water look the project should try to model water rebounding off the edges of the pool.

This could be modelled by improving the current model for water movement discussed in 6.2.1. In the

current water model, the direction of wave movement is captured as a 2D vector Di . This vector would

need to be re-evaluated for every wave every frame to test for collision detection against the water edge.

Adding this element to the model would improve the realism of the water scenes by addressing the negative

comments regarding the directional flow of the water.

There are two possible ways to increase the randomness of the ripples in the water. A simple

approach would be to include more waves in our current model for water movement. This is a simple

addition and was attempted to try and improve the look. Initial tests revealed that adding more waves greatly

slowed the rendering speed of the water. The decision was made to abandon this approach because the slow

down was so severe. Another possible improvement to the water movement was to adopt another model of

water, namely Gerstner waves (Finch, 2004). This model alters the vertex position in 3 dimensions as

opposed to only the height dimension. This greatly improves the rolling appearance of waves. This

improvement was explored and a video of the result can be found at http://www.youtube.com/watch?

v=xLg6q_bZ_M8 . Unfortunately, there was not enough time to add the lighting conditions to the scene,

therefore the rendering speed of this new model could not be calculated; this is why the model is displayed as

a wire frame.

48

7.5 Conclusion and Graphics Turing Test

Overall, the evaluation component to this project brought to light some useful information regarding the

mathematical models that were used to implement the scenes in this project. The fact that the public labelled

the majority of the videos as almost convincingly real shows real promise for real-time photo realistic

computer graphics. One can learn from the public response and, given more time, further improve the

realism of the videos by listening to the critical feedback. Perhaps one day a project will be able to conclude

with 100% realism and succeed in implementing some truly outstanding photo-realistic computer graphics.

Taking this into account, this project will conclude with a challenge. A computer graphics version of the

Turing Test in artificial intelligence. If any party can follow the same methods as this project and score

100% for realism and 100% for the match description then they will pass the test. It is open to anyone who

wishes to try. See Wells (2009) for more information.

49

Bibliography

Astle, “More OpenGL Game Programming”, Thomson course technology ISBN-10: 1592008305, 2006.

Boehm, "A Spiral Model of Software Development and Enhancement", "Computer", "IEEE", 21(5):

pp.61-72, 1988

Bucanac, “The V Model”, 1999 http://www.bucanac.com/documents/The_V-Model.pdf [accessed

04/12/2008].

Bungie, http://www.bungie.net/Inside/publications.aspx#pub13318, 2008 [accessed 07/04/2009].

Cook and Torrance, “A REFLECTANCE MODEL FOR COMPUTER GRAPHICS”, Computer Graphics

Volume 15, Number 3, 1981.

Crytek, http://www.crytek.com/technology/cryengine-2/specifications/ , 2008, [accessed 01/12/2008].

d'Eon and Luebke, “Advanced Techniques for Realistic Real-Time Skin Rendering” in GPU Gems 3, Nvidia

chapter 14, pp. 293-347, 2007.

50

Duff and Porter. "Compositing Digital Images". Computer Graphics 18 (3): pp. 253–259, 1985.

Finch, “Effective Water Simulation from Physical Models” in GPU Gems chapter 1,

http://http.developer.nvidia.com/GPUGems/gpugems_ch01.html ,2004 [accessed 06/12/2008].

Foley, James, Andries van Dam, Feiner, and Hughes, “Computer Graphics: Principles and Practice, 2nd ed.”

Addison-Wesley, 1996.

Frisvad, Christensen, Jensen, "Computing the Scattering Properties of Participating Media using Lorenz-

Mie Theory" ACM Transactions on Graphics SIGGRAPH , 2007.

Gouraud, "Continuous shading of curved surfaces," IEEE Transactions on Computers, 20(6): pp. 623–

628, 1971.

Green, http://www.research.scea.com/gdc2003/spherical-harmonic-lighting.pdf, 2003 [accessed 07/04/2009].

Guardado, “Rendering Water Caustics“ in GPU Gems chapter 2, http://http.developer.nvidia.com/GPUGems/

gpugems_ch02.html ,2004 [accessed 06/12/2008].

Immel, David S.; Cohen, Michael F.; Greenberg, Donald P., "A radiosity method for non-diffuse

environments", SIGGRAPH 1986.

Kajiya, James T., "The rendering equation", SIGGRAPH 1986.

Kautz, Sloan, Snyder “Fast, Arbitrary BRDF Shading for Low-Frequency Lighting Using Spherical

Harmonics“ in Proceedings of the 13th Eurographics Workshop on Rendering , Pages: 301-308, 2002.

Luna, “Introduction to 3D Game Programming with Direct X 9.0c: A Shader Approach”, Wordware Game

and Graphics Library ISBN-10: 1598220160, 2006.

Microsoft, http://msdn.microsoft.com/en-us/library/bb147327(VS.85).aspx , 2008 [accessed 6/12/2008].

Microsoft, http://msdn.microsoft.com/en-us/library/bb173329(VS.85).aspx, 2008b [accessed

6/12/2008].

Minnaert, “The Reciprocity Principle in Lunar Photometry” Astrophysical Journal, vol 93, 1941.

51

Nayar and Oren, "Generalization of the Lambertian Model and Implications for Machine Vision".

International Journal on Computer Vision, Vol.14, No.3, pp.227-251, 1995.

Phong, “Illumination for computer generated pictures” in Communications of the ACM archive, Volume

18, Issue 6, Pages: 311 – 317, ISSN:0001-0782, 1975.

Royce, “MANAGING THE DEVELOPMENT OF LARGE SOFTWARE SYSTEMS” Proceedings, IEEE

WESCON, pp 1-9, 1970.

Tessendorf, “Simulating Ocean Water”, http://graphics.ucsd.edu/courses/rendering/2005/jdewall/tessendorf.pdf , 2004, [accessed 06/11/2008].

Turing, “Computing machinery and intelligence”. Mind, 59, pp.433-460, 1950.

VanVerth and Bishop, “Essential Mathematics for Games and Interactive Applications”, Elsevier Science &

Technology Morgan Kaufmann Publishers, ISBN10: 0123742978, 2008 .

Wells “The Graphics Turing Test”, http://www.benwells3d.com/GTT.pdf 2009, [accessed 27/04/2009].

Zerbst and Duvel, “3D Game Engine Programming”, Thomson Course Technology, 2004.

52

Appendix A

This was a challenging project, but with challenge comes reward and the reward was very great

indeed. I had to learn a lot of new maths and programming skills in order to meet the minimum requirements

for this project. The journey was long and difficult but very satisfying and I now feel that I have built a

sufficiently large toolbox of skills and techniques in computer graphics to perform well in the computer

games industry; my next step after university.

I found that the project came together well due to a good schedule and methodology. It is very

tempting to put parts of the project off until a later date but beware, this is the first step to a failed project.

My advice would be to stick to the schedule, this is the most important advice I can give to a student

embarking on their final year project.

I am pleased that I chose such a challenging project and I invite any student to try and push

themselves by picking a project with similar difficulty. If any student wishes to do so then my advice would

be to choose the minimum requirements very carefully. Challenging projects come with the added risks, the

project may become too challenging so take this into account when deciding on the minimum requirements.

Try to lower the risk by deciding the minimum requirements after you have researched the techniques and

skills needed, not before. I was surprised to hear that some fellow students had chosen their minimum

requirements before knowing how they would go about meeting them. This is a bad idea and should be

avoided.

Upon reflection of project direction I would advise fellow students to pick a project in a subject area

53

that they are interested in and try their utmost to harmonise it with their interests of the assessor. Students

know who their assessor is from a very early stage in the project so this is achievable and something that I

took into account when writing this report; I concentrated a lot on the physics of my scenes and less on the

rendering engine architecture, for example.

Another piece of advice I would direct to students who are about to undertake a project such as this

is to know the overall direction of the project by Christmas time. Do not leave it any later. This piece of

advice especially applies to students who decide their own project, like myself, because by Christmas you

have to submit a mid project report that has to convince your assessor and supervisor that your idea

constitutes a final year project. Therefore, if you do not know the direction of the project by then, you will

not be able to convince them.

My last piece of advice would be that if you pick your own project then make sure that you have

enough self drive to carry the project through to the end because you may have to tackle problems with less

guidance from your supervisor than if you had picked your project from a list.

54