Simulating the Effect of Illumination Using ColorTransformations

Maya R. Guptaa and Steve Uptonb and Jayson Bowena

aDept. of Electrical Engineering, University of Washington, Seattle, WAbChromix, Seattle, WA

ABSTRACT

We investigate design and estimation issues for using the standard color management profile architecture forgeneral custom image enhancement. Color management profiles are a flexible architecture for describing amapping from an original colorspace to a new colorspace. We investigate use of this same architecture fordescribing color enhancements that could be defined by a non-technical user using samples of the mapping, justas color management is based on samples of a mapping between an original colorspace and a new colorspace. Asan example enhancement, we work with photos of the 24 color patch Macbeth chart under different illuminations,with the goal of defining transformations that would take, for example, a studio D65 image and reproduce it asthough it had been taken during a particular sunset. The color management profile architecture includes a look-up-table and interpolation. We concentrate on the estimation of the look-up-table points from minimal numberof color enhancement samples (comparing interpolative and extrapolative statistical learning techniques), andevaluate the feasibility of using the color management architecture for custom enhancement definitions.

Keywords: ICC profile, custom color enhancement, LIME, color mapping, 3D LUT

1. INTRODUCTION

Digital design and graphics arts is an expanding field, due in part to cheap and easy access to high-qualitycolor printing technologies and the growth and commercialization of the Internet. Design plays an indispensablerole in perceptions of quality, design, and branding1.2 How do we empower digital designers to create customcolor enhancements? Recently, there has been interest in computer-aided color image enhancement that allowsusers to create complicated or custom effects with limited work or technical knowledge. This paper explores thestatistical learning issues that arise in learning a custom color enhancement based on a small number of sampleinput/output color pairs. In particular, we use the ICC profile architecture to define and implement a customtransformation. As an example application, this paper considers the problem of simulating illumination effectsby a color transformation.

ICC profiles3 were designed for color management of devices, such as printers, which require empirical char-acterization and complex color transformations to model their sampled characteristics. The core of an ICCprofile is a three-dimensional look-up-table (LUT) that defines how input colors are transformed into outputcolors. There are two major advantages to making a custom color enhancement into an ICC profile. First, athree-dimensional look-up-table is a very flexible way to define a color transformation. Second, once defined,the color transformation can be implemented by any color management module. This allows users to implementtransforms using non-proprietary software and view and edit transforms using color management applicationslike ColorThink.4 ICC profiles are standardized by the International Color Consortium to provide an open,vendor-neutral device characterization for color management.

Simulating illumination changes is an example of a complex color transformation that can be defined bysample pairs. For example, a designer might wish that a studio photo of a hamburger or model was taken underthe light of a certain Namibian sunset. If the designer has captured the effect of that Namibian sunset and theeffects of the studio light, then the effect of the sunset can be simulated on the studio photo. In this paper,

Further author information: (Send correspondence to M.R.G.)M.R.G.: E-mail: gupta@ee.washington.edu,S. U.: E-mail: upton@chromix.com

we capture the effect of an illuminant on the 24 color samples of a Gretag Macbeth ColorChecker chart, andthen “learn” a color transformation for the entire colorspace based on input/output pairs of, for example, D65illumination to a particular sunset light.

2. RELATED WORK

Aiding designers to produce complex custom color enhancements is a popular area of research, and we referenceonly a few representative examples. One method to define a custom enhancement is to transform the color paletteof an input image based on the color palette of a single reference image5.6 In such work, the output imagespatially looks like the input image, but has inherited or moved towards the color palette from the referenceimage. Hertzmann et al.7 use a pair of reference images to learn a transformation that can then be applied to athird image. They use multi-scale spatial features as well as pixel color values to create image transformationssuch as adding texture.

LUT’s have previously been used to speed-up known functions that transform colors or images.8 In Gattaet al.’s opinion,9 “The LUT transfer function is the simplest way to apply a global filtering to an image usinga function previously devised.”

In contrast to prior work, this paper confines itself to colorspace transformations that can be implementedwith a three-dimensional LUT, and that are not already implementable by a known function or algorithm, sothat the amount of data to learn from is relatively small sample of input/output color pairs (< 100), as mightbe defined by a graphic artist. This paper explores the use of statistical learning to create an accurate andvisually smooth three-dimensional LUT transformation. We use simulating illumination changes as an exampleapplication and data source.

Another way to simulate illumination is to estimate the reference and destination illumination spectra, thendivide out the reference spectra from the three color channels of the image, and multiply-in the destinationillumination spectra. Direct illumination spectra estimation has been studied in work such as the scannercalibration literature. Since we are interested in an architecture for general color enhancement based on inputand output samples, illumination spectra estimation is not general enough for our goals. Whether simulatingthe effect of an illuminant by estimating and applying the spectra, or as we do in this paper, by learning a colortransformation based on samples, there remains the problem of metamers. Objects originally photographed thatappear the same in the image may have physical properties that would make them not look the same underdifferent illumination. For example, water and jeans might be metameric objects, both appearing the same colorof blue in the image to be transformed. Without semantic knowledge of what the object is, all pixels of a certaincolor value will be transformed in the same way by the proposed color transformations. Work on semanticobject recognition,10 may one day be coupled with object specific transforms, but in the present work we ignoremetameric complications.

This work is also related to white balance for digital cameras. White balance algorithms are generally simpleand serve a different purpose than the enhancements investigated here.

3. ILLUMINANT DATA

Photographs of a standard 2004 Gretag Macbeth ColorChecker chart under different illumination conditionswere used as the data for this project. A Sony DSC-F828 8 MegaPixel camera was used. For each picture, thecolorchart and a standard photographer’s gray card were set-up on a board. The camera was set to manualmode with the white balance set to daylight. The metering mode was set to “spot” and the angle of the boardwas adjusted until all the readings on the gray card were equal. Then the photo was taken. Example photos areshown in Fig. 1. A D65 photo was taken of the chart under a Gretag Macbeth SolSource D65 filtered lamp. Foreach of the twenty-four color squares in each image, the corresponding sRGB pixel values from the camera wereaveraged, forming a data set of twenty-four sRGB values for each illumination condition. The sRGB values arethen converted to CIELab values (D65 whitepoint) by standard formula.

Differences in CIELab between pairs of illuminant samples are shown in Fig. 2.

Figure 1. Left: chart photographed under cloudy conditions. Right: chart photographed under a soft white incandescentlight.

L

a

b

L

ab

Figure 2. Left: vectors show the differences between the 24 D65 color chart samples and 24 soft white incandescent colorchart samples in three-dimensional CIELab space. Right: vectors show the differences between the 24 cloudy color chartsamples and 24 soft white incandescent color chart samples in three-dimensional CIELab space.

4. ESTIMATING THE 3D LUTBased on the known input/output CIELab color sample pairs, a 3D LUT is estimated which maps a 21x21x21grid in CIELab to corresponding output CIELab color values. Estimating the 3D 21x21x21 grid and its ouputvalues from only 24 sample pairs is a difficult learning problem, particularly because many of the grid pointsfall outside the convex hull of the set of 24 samples in the input space. The estimation issues are interpolationvs. extrapolation, a flexible enough fit to capture the underlying transform vs. overfitting the transform, andsmoothness and monotonicity of the estimated transform. Because this is a color enhancement, smoothness ofthe transform around the neutrals is particularly important. In this paper we consider two radically differentmethods for estimating the 3D LUT grid points, local linear regression, and linear interpolation with maximumentropy.

Regression methods can be categorized as interpolative or extrapolative. For example, local linear regression11

on a k-neighbor neighborhood fits a plane to the k samples nearest the grid point in the input space out of thetotal 24 original input samples. Since a plane is fit to the samples closest to each grid point, the methodextrapolates to grid points that are outside the convex hull of the set of 24 samples in the input space. Locallinear regression has been shown to have nice properties near boundaries and for unevenly distributed samplepoints,12 as is the case for this application.

Consider, on the other hand, common tetrahedral interpolation sometimes used in color management,13

which linearly interpolates the four samples closest to a particular grid point in the input space in order toestimate that grid point’s output value. If a grid point is outside of the convex hull formed by its four nearestsamples, then tetrahedral interpolation weights cannot be solved for, since the weights must sum to one.

Let x be a point on the grid that forms the 3D LUT. Let y be the estimate of its transformed color. Letx1, x2, . . . , x24 be the 24 sample points in the input colorspace, where they have been indexed by their Euclideandistance to x in the input CIELab space. Let y1, y2, . . . , y24 be the corresponding output, or transformed, CIELabcolor values.

Then tetrahedral interpolation calculates weights wj that solve,∑4

j=1 wjxj = x

subject to4∑

j=1

wj = 1

wj ≥ 0 for all j .

(1)

Once the tetrahedral weights w1, w2, w3, w4 have been found (if they exist), the tetrahedral interpolation estimateis

y =4∑

j=1

wjyj

As noted above, (1) cannot be solved for all grid points, tetrahedral interpolation can only be used if the gridpoint x being estimated is in the closure of the convex hull formed by its four nearest-neighbors.

Linear interpolation with maximum entropy (LIME)14 generalizes linear interpolation so that any number ofsample points can be used and there is no problem if the test point x lies outside the convex hull of its neighbors.The LIME weights attempt to solve the linear interpolation equations. If x is not in the convex hull of itsneighbors, then the LIME weights attempt to minimize the distance between the reconstructed point

∑j wjxj

and x, which in effect projects x onto the convex hull of the neighbors. If the linear interpolation equations areunderdetermined (as may be when k > 4 for three dimensions), then there exist multiple solutions to the linearinterpolation equations, and the LIME weights are the solution that maximizes entropy. Formally, the LIMEweights wj solve,

arg minw

(‖∑k

j=1 wjxj − x‖2 + λ∑k

j=1 wj log wj

)

subject tok∑

j=1

wj = 1

wj ≥ 0 for all j,

where λ and k are algorithmic parameters. In this work, we set λ = 1e − 9, so that the weights focus onminimizing the distortion between

∑j wjxj and x. For k = 4, the LIME weights will be numerically equivalent

to the tetrahedral weights, if the tetrahedral weights exist. Given the LIME weights wj , the color value estimatedto be the output corresponding to input grid color x is,

y =k∑

j=1

wjyj (2)

5. RESULTS

For different pairs of input/output illumination conditions, we compare local linear regression and LIME estimatesof the 3D LUT by comparing how new points are transformed when the estimated 3D LUT transformationis applied. To estimate a new color value z, the eight vertices of its grid cell are determined, and then z’scorresponding output color is calculated using trilinear interpolation.13

First, we consider a test case of neutral colors. After estimating the 3D LUT for transforming D65 illuminationto a sunset illumination, we transform the neutral axis in CIELab using the estimated 3D mappings. The resultsare shown in Fig. 3. It is clear from the plots that the estimation algorithm and the number of neighbors usedsignificantly changes the estimation, even in the center of the colorspace. As expected, increasing the numberof neighbors leads to a smoother fit, but may smooth out the desired color mapping. Using four near-neighborswith the local linear regression leads to a non-monotonic luminance transformation of the neutrals axis, andlarge hue variance in the mid-grays. Overall, the LIME estimate is less variable with respect to the numberof neighbors used. The LIME change in hue and luminance is also smoother over the neutral axis for a givennumber of neighbors. However, some of that smoothness is due to the interpolative nature, which limits thedarkest dark and brightest white to the convex hull of the original twenty-four samples.

What happens on the edge of the D65 to sunset mapping? In Fig. 4, the output CIELab color valuescorresponding to the regular 21 × 21 × 21 input CIELab grid are shown for the grid estimated by LIME andthe grid estimated by local linear regression, each using six nearest-neighbors. Here, the difference between theinterpolative and extrapolative method is clear. Since it is doing only interpolation, the LIME grid will map allinput colors to a small gamut as defined by the convex hull of the original twenty-four samples. This can leadto images where very saturated object regions are clipped and appear flat.

The local linear regression grid maps input colors to the entire space; in fact the colors shown have beenclipped to be within L ∈ [0, 100] and a, b ∈ [−100, 100]. Many of the colors that the local linear regression mapsto will be outside the gamut of standard monitors or printers, and thus many points will again be clipped upondisplay, leading to some flatness in extreme color regions of an image. Because the local linear regression gridextrapolates to the entire space, the average distance between output colors is much greater than the averagedistance between output colors from the LIME grid. This can lead to false edges in local linear regression mappedimages.

In Fig. 5, a daylight photograph of a building (from the Foveon Image Gallery) is transformed using aD65 to sunset color transformation. The left transformations are with local linear regression, and the righttransformations are with LIME estimation. In the context of the image, the transformations look quite similar,with the most noticeable differences occurring in the brightest areas.

In Fig. 6, a daylight photograph of a landscape is transformed from daylight (D65) to sunset using fourdifferent transformations. In both Fig. 5 and Fig. 6 the brightest regions of the local linear regression may strikesome viewers as “too bright” compared to the rest of the image’s palette.

In Fig. 7, a clearer difference between the estimation methods is seen in this application of a “soft whiteincandescent” to “cloudy” color transformation. The specular highlights on the red pepper have turned blue inthe LIME images, and this will strike most viewers as unnatural. Though the off-color specular highlights area very tiny portion of the image, they give the sense that the entire image is too blue, due to human’s extremesensitivity to highlights. Less offensive, but also unnatural, is the flat bright white region of the foregroundgarlic in the local linear regression images, which is inconsistent with the rest of the palette. Though not too

L L

L L

L L

Figure 3. Plots show how the neutral CIELab axis is transformed under the estimated color mapping for D65 illuminationto a sunset illumination. Left: local linear regression using 4, 6, and 9 neighbors (from top to bottom). Right: LIMEusing 4, 6, and 9 neighbors (from top to bottom).

L

L

Figure 4. Both plots show the 3D grid mapping from D65 to a sunset illumination. Left: local linear regression outputvalues corresponding to the 3D LUT grid points, using a neighborhood of six of the twenty-four original input samples.Right: LIME output values corresponding to the 3D LUT grid points, using a neighborhood of six of the twenty-fouroriginal input samples.

disagreeable, the extrapolated local linear regression highlights and bright areas seem again “too bright” forthe color cast of the rest of the image. The differences between the six and nine neighbor images are not verynoticeable in the images shown.

6. DISCUSSION

In this paper we have investigated issues arising from estimating color transformations using a 3D LUT basedon relatively few samples. We have shown that either local linear regression or LIME estimation can result insensible transformations for realistic image conversions. An important estimation issue for this application is theinterpolative vs. extrapolative nature of the estimation. Interpolation is severely limited by its very nature; forneutral images the interpolation fit may better reflect the desired transformation, but for images with extremecolor values any interpolative method will lead to clipping and objects can acquire an appearance of flatness dueto lost contrast. The clipping due to interpolation can also cause miscolored specular reflections to which theeye is extremely sensitive.

Other regression or smoothing algorithms could be used to estimate the 3D LUT. Due to the sparsity ofdata samples, polynomial or other spline approaches may have a negative extrapolative effect, resulting in highvariance of the estimated color transformation for small changes in algorithm parameters. Further, extrapolativeestimation risks assigning extreme output color values and failing to consistently capture the sense of the desiredtransformation.

The key issue for this application is finding a middle way between interpolating and extrapolating that retainsthe sense of the transformation, and optimally fills the output colorspace, resulting in minimal clipping. Further,since the goal is an automated system that does not require the user to carefully analyze different estimatedtransformations, the estimation method should lead to a sensible transformation for a large class of possibleoriginal sample pairs. This goal is more likely to be achieved with estimation methods that are robust, in thesense that small changes in the estimation parameters (such as changing the neighborhood size) lead to smallchanges in the estimated transformation. Such robustness will help ensure consistently reasonable estimationbehavior over a large class of input sample pairs.

Figure 5. Top left: original daylight (D65) image. Left: original image transformed using D65 to sunset mappingestimated with local linear regression (middle with six neighbors and bottom with nine neighbors). Right: original imagetransformed using D65 to sunset mapping estimated with LIME regression (middle with six neighbors and bottom withnine neighbors).

Figure 6. Top left: original daylight (D65) image. Left: original image transformed using D65 to sunset mappingestimated with local linear regression (middle with six neighbors and bottom with nine neighbors). Right: original imagetransformed using D65 to sunset mapping estimated with LIME regression (middle with six neighbors and bottom withnine neighbors).

Figure 7. Top left: original image taken under unknown lighting. Left: original image transformed using soft whiteincandescent to cloudy mapping estimated with local linear regression (middle with six neighbors and bottom with nineneighbors). Right: original image transformed using soft white incandescent to cloudy mapping estimated with LIMEregression (middle with six neighbors and bottom with nine neighbors).

Defining color image enhancements via 3D LUT’s based on a small set of sample pairs is an interesting andeasy way to create custom image enhancements. The statistical estimation used must be robust for this to beuseful in practice.

REFERENCES1. B. H. Schmitt and A. Simonson, Marketing Aesthetics: The Strategic Management of Brands, Identity and

Image, Free Press, 1997.2. V. Postrel, The Substance of Style: How the Rise of Aesthetic Value Is Remaking Commerce, Culture, and

Consciousness, Harpers-Collins, 2003.3. “ICC webpage,” 2004. www.color.org/profile.html.4. “ColorThink 2.1.2,” 2004. www.chromix.com.5. Y. Chang and S. Saito, “Example-based color stylization based on categorical perception,” Proc. of the First

Symposium on Applied Perception in Graphics and Visualization , pp. 91–98, 2004.6. E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley, “Color transfer between images,” IEEE Computer

Graphics and Applications 21, pp. 34–41, September 2001.7. A. Hertzmann, C. E. Jacobs, B. C. N. Oliver, and D. H. Salesin, “Image analogies,” SIGGRAPH , pp. 327–

340, 2001.8. D. M. A. Rizzi and L. D. Carli, “Multilevel brownian retinex colour correction,” Machine Graphics and

Vision , 2002.9. D. M. A. R. C. Gatta, S. Vacchi, “Proposal for a new method to speed up local color correction algorithms,”

Proc. of the SPIE 5293.10. Y. Li, J. A. Bilmes, and L. G. Shapiro, “Object class recognition using images of abstract regions,” Proc.

of the 17th Intl. Conf. on Pattern Recognition , 2004.11. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, Springer-Verlag, New York,

2001.12. T. Hastie and C. Loader, “Local regression: automatic kernel carpentry,” Statistical Science 8(2), pp. 120–

143, 1993.13. H. Kang, Color Technology for Electronic Imaging Devices, SPIE Press, United States of America, 1997.14. M. R. Gupta, “Inverting color transformations,” Proceedings of the SPIE Conf. on Computational Imaging

, 2004.