Automatic Color Realism Enhancement for Virtual RealityHyunjung Shim∗

SAIT, Samsung ElectronicsSeungkyu Lee

SAIT, Samsung Electronics

ABSTRACT

Photorealism has been one of essential goals for virtual reality. Thestate-of-the-art techniques employ various rendering algorithms tosimulate physically accurate light transport for generating the pho-torealistic appearance of scene. However, they still require a labor-intensive tone mapping and color tunes by an experienced artist. Inthis paper, we propose an automatic photorealism enhancement al-gorithm by manipulating the color distribution of graphics to matchwith that of real photographs. Based on the hypothesis that photore-alism is highly correlated with the frequency of color characteristicsappearing in real photographs, we find principal color components.Then, we transfer the statistical characteristics of photographs ontographics so to enhance their photorealism. Experiments and a userstudy have confirmed the effectiveness of proposed method.

1 INTRODUCTION

Photorealistic computer generated imagery has been a long-lastinggoal for virtual reality. For last few decades, extensive studieshave been conducted for physically accurate rendering in hopesof achieving photorealism comparable to photographs. However,the state-of-the-art techniques by film and television industry stilluse labor-intensive fine tunes and modifications afterwards for fi-nal production. In this paper, we focus on colors in images as animportant factor to govern the photorealism.

Existing studies consistently report that colors significantly af-fect the photorealism [3]. In fact, we often find that color selectionsand fine tunes by an experienced artist is crucial to finalize the qual-ity of graphics contents. This empirically justifies an important roleof colors in images. Therefore, we investigate principal color com-ponents for photorealism and apply our findings onto synthesis todevelop an automatic photorealism enhancement technique. Morespecifically, we collect two classes of images from the web: approx-imately 60,000 of graphics (computer generated) and photographs(real) for each. Then, we analyze the color statistics of each classand learn how to manipulate the colors of graphics so to match withthat of photographs. This prior knowledge is used to enhance thephotorealism of arbitrary graphics.

To the best of our knowledge, we propose the first computationalapproach for automatic photorealism enhancement. Moreover, ourapproach results some interesting outcomes that support the psy-chophysical study conducted by color scientists; the average re-flectance in a scene under a neutral light source is achromatic [1].We believe that it is meaningful to confirm an analogy between psy-chophysical study and our data-driven computational approach.

2 ALGORITHM DESCRIPTION

The overview of our approach is illustrated in Fig. 1. The proposedapproach consists of two stages: training two-class color priors us-ing a collection of images and enhancing the photorealism of aninput image using an energy minimization technique. These twostages constitute our photorealism enhancement technique. To gain

∗e-mail: kate.shim@samsung.com

the credibility of the proposed approach, we perform extensive ex-periments and a user study to validate the effectiveness of our ap-proach.

Data-Driven Color Priors. We construct two-class color distri-butions using images collected from the web. For building the dis-tributions, we define the feature of distribution, the principal colorcomponent (PCC) of each image, by satisfying following two ob-jectives: 1) representing the color distribution of one class well and2) distinguishing one class from another. For that, we calculate theeigenvectors, the eigenvalues of its covariance matrix and the meanvector for each image, denoted by e, λ and m respectively. ThesePCA results consist of our PCC for one image. Throughout thispaper, we use u = [e1;e2;e3;λ1;λ2;λ3;m] to denote a PCC for oneimage. Finally, we extract PCCs for all images per class to formthe distribution of PCCs. Then, we apply the support vector ma-chine for separating two classes and use a set of support vectors toconstruct the representative color distribution of each class.

Photorealism Enhancement by Energy Minimization. We in-troduce photorealism enhancement based on an energy minimiza-tion technique. Given by an input image (arbitrary graphics), itsPCC is extracted. Then, we subsample the entire color distributionsinto their local distribution, only including samples closely locatedto the input for each class. We measure the distance of input PCCfrom each class, translating into the dissimilarity of the input fromeach class. In order to enhance the photorealism, we manipulatethe input by positioning it 1) close to the class of photograph, 2) faraway from the class of graphics and 3) closer to the original posi-tion in color domain. We formulate this problem by minimizing thesummation of three energy terms. We define the energy function asfollows:

E = αE1 +βE2 + γE3, (1)

where E1 = 1Np

∑i∈Np ‖Fp(i)− u‖2, E2 = − 1Ng

∑i∈Ng ‖Fg(i)− u‖2

and E3 = ‖p− p‖2. u, Fp and Fg denotes the PCC of input im-age, the color prior of photographs and the color prior of graphicsrespectively. u is our estimated PCC of input image. Np or Ng de-fines a set of the neighboring samples in Fp or Fg with respect tothe input u. Np or Ng is a total number of samples in Np or Ng. Wecalculate u so to minimize the energy function in Eqs 1.

E1 is the first energy term that indicates the distance betweenthe input and the color prior of photographs. By minimizing thisenergy term, we can locate our estimate close to the color prior ofphotographs. In our application, we decide to only consider theneighboring samples because such a local distribution is more in-formative to guide the optimization. The entire distribution of Fpspans a wide range of images and some samples of Fp may not berelevant to u. To define a set of neighborhood Np, we use a pre-determined distance Dp = 8 or increase Dp until Np is over 100. E2defines how close the input is located from the color prior of graph-ics. This second energy term positions our estimate u far from Fgso that the undesired color characteristics can be vanished. Again,we set Dg = 8 or increase Dg until Ng is over 100. E3 is consid-ered to be a regularization constraint, which is the distance betweenu and u. We minimize this energy term so to change the originalcolor distribution as little as possible while still achieve the effectof photorealism enhancement. This regularization constraint alsomakes the estimation reliable. Finally, we compute for u by solving

59

IEEE Virtual Reality 20124-8 March, Orange County, CA, USA978-1-4673-1246-2/12/$31.00 ©2012 IEEE

Figure 1: Overview of the proposed method. Stage 1: Training two-class color priors using tens of thousands of images collected from the web,Stage 2: Photorealism enhancement by an energy minimization technique, PCC: principal color components, SVM: support vector machine.

the energy minimization problem defined by Eqs. 1.Given the estimate of PCC, we modify the color distribution of

input image for photorealism enhancement. The original color dis-tribution of input image can be decomposed by the weighted sumof eigenvectors and can be expressed as:

[i1 . . . iM ] =[

e1 e2 e3] w1

1 . . . w1M

w21 . . . w2

Mw3

1 . . . w3M

+m1 (2)

i and 1 represent a color vector (3× 1) and a one vector (1×M).Also, M is a total number of pixels in the input image and w is acoefficient corresponding to each eigenvector and each pixel. Bysolving the energy function in Eqs. 1, we solve for u. Then, weapply the estimated PCC onto the entire color distribution by sub-stituting u by u. To reflect the change in λi, we multiply λi/λi onthe coefficients.

3 RESULTS AND ANALYSIS

In this section, we present the results of experimental evaluation.First, we demonstrate various results to show the performance ofproposed algorithm for enhancing the photorealism. From the re-sults in Fig. 2 (a), we observe that our method manipulates theentire color tone so that the resultant scene appears as if it is illu-minated by the white light. This is analogous to the Grey WorldAssumption [1]; the average reflectance in a scene under a neutrallight source is achromatic. Note that the over saturation and exag-geration in colors are the crucial difference between graphics andphotographs [2]. We remove an excessive color components in im-age for photorealism enhancement and this tendency often appearsin our results. Yet, our operation does not simply make an entireimage achromatic. The results of Fig. 2 (b) show that our operationincreases the chroma of the partial region in the input image; redchair, blue marbles and red door. While our operation introducesthe strong contrast and colorfulness in Fig. 2 (b), it still changesthe ambient of the input image as if it is lit by the natural whitelight. We consistently observe the similar phenomena from variousexperimental results.

Second, we conduct a user study for the quantitative evaluationof proposed method. This user study is conceived by measuringour achievement in photorealism, compared with an original im-age. We use 75 test images, computer-generated images spanningdifferent types of scenes. We ask 30 participants to assign the score1) 1 if the image looks more realistic than another, 2) −1 if it looksless realistic than another and 3) 0 if it is hard to determine. Eachparticipant evaluated a total of 20 images, randomly drawn from75 images. We perform a T-test on our results to see how signifi-cant improvement has been accomplished. Our result on T-test re-jects the null hypothesis with a positive T value (7.09) and a smalltwo-tailed P value (much less than 0.05). This means that we have

(a) Various scenes

(b) Various color enhancing effects

Figure 2: Photorealism enhancement. Top: Original graphics, Bot-tom: Our results.

achieved the statistically meaningful improvement in photorealismenhancement.

4 CONCLUSION AND DISCUSSION

We have presented an automatic photorealism enhancement tech-nique. Proposed algorithm shows a great potential to automate la-bor intensive color tunes, powered by data-driven color priors. Weexpect that the proposed work can be applied onto the renderedframe for increasing the photorealism. By conducting various ex-periments and a user study, we have justified the effectiveness ofour method.

REFERENCES

[1] G. Buchsbaum. A spatial processor model for object colour perception.Journal of the Franklin Institute, 310:1–26, July 1980.

[2] M. K. Johnson, K. Dale, S. Avidan, H. Pfister, W. T. Freeman, andW. Matusik. Cg2real: Improving the realism of computer generatedimages using a large collection of photographs. IEEE TVCG, 99:1–13,2010.

[3] J.-F. Lalonde and A. A. Efros. Using color compatibility for assessingimage realism. IEEE International Conference on Computer Vision,2007.

60