A New In-Camera Imaging Model For Color Computer Vision And Its Application

A New In-Camera Imaging Model for

Color Computer Vision and its Application

Hai Ting LIN Supervisor: Prof. Brown

National University of Singapore

May 9th, 2012

Joint work with: Seon Joo Kim, Lu Zheng, Sabine Süsstrunk, Stephen Lin

Talk overview

• Motivation – Modern digital cameras

– Cameras and computer vision

• Our work (ICCV’11) – A new imaging pipeline for cameras

• Application of our pipeline – sRGB to RAW

– Photo refinishing (and back)

Camera = light-measuring device

From Digital Image Processing, Gonzales/Woods

Simple models assume an image is a measurement of scene radiance.

Illumination source (Scene radiance)

Internal Image Plane

Scene Element

Imaging System

Output (digital) image

Camera = light-measuring device

•Shape from shading •Intrinsic image •Image Matching •HDR Imaging •Etc . . .

Shape-from-shading

Image Matching

Lu et al, CVPR’10

From Jon Mooser, CGIT Lab, USC From O’Reilly’s digital media forum

HDR Imaging

Light-measuring device?

Onboard Photofinishing “Secret Recipe” of a Camera

Three different cameras with same aperture, exposure, white-balance and picture style, etc. . .

Can image values be treated as physically meaningful values?

And if so, when and how?

Modern photography pipeline

8

In-Camera CCD response (RAW)

CCD Demosaicing (RAW)

“Photo-finishing Processing”

Scene Radiance

Starting point: reality (in radiance)

Camera Output: sRGB

Pre-Camera Lens Filter

Lens Shutter

Aperture

Digital camera pipeline (early work)

Debevec and Malik *SIG’97+

f (x) E*k (RAW)

Unknown f . . . the camera’s non-linear response to RAW.

“Radiometric Calibration”

Accepted model

bx

gx

rx

E

E

E

bx

gx

rx

E

E

E

T

bx

gx

rx

e

e

e

=

)(

)(

)(

bxb

gxg

rxr

ef

ef

ef

bx

gx

rx

i

i

i

=

(RAW) T is a 3x3 matrix i is the sRGB output and f is a non-linear function

Related work

Mann and Picard, SPIE’95 Debevec and Malik, SIG’97 Mitsunaga and Nayar, CVPR’99 Farid, TIP’01 Grossberg and Nayar, TPAMI’03 Grossberg and Nayar, TPAMI’04 Lin et al, CVPR’04 … Manders et al, ICIP’04 Pal et al, CVPR’04 Lin et al, ICCV’05 Kim and Pollefeys, TPAMI’08

Fixe

d p

rop

erty

of

the

cam

era

)(

)(

)(

1

1

1

bxb

gxg

rxr

if

if

if

bx

gx

rx

e

e

e

=

Related work

Mann and Picard, SPIE’95 Debevec and Malik, SIG’97 Mitsunaga and Nayar, CVPR’99 Farid, TIP’01 Grossberg and Nayar, TPAMI’03 Grossberg and Nayar, TPAMI’04 Lin et al, CVPR’04 … Manders et al, ICIP’04 Pal et al, CVPR’04 Lin et al, ICCV’05 Kim and Pollefeys, TPAMI’08 Chakrabarti et al, BMVC’09

Chakrabarti et al conclusions: RAW is meaningful . . . . But, requires a 24 parameter model that is scene-dependent to accurately go back from sRGB to RAW.

Fixe

d p

rop

erty

of

the

cam

era

)(

)(

)(

1

1

1

bxb

gxg

rxr

if

if

if

bx

gx

rx

e

e

e

=

bx

gx

rx

E

E

E

T

bx

gx

rx

e

e

e

=

)(

)(

)(

bx

gxg

rxr

ef

ef

ef

bx

gx

rx

i

i

i

=

Accepted model

Is this model good enough?

Is processing scene dependent?

Observations

Where are the outliers?

Color rendering

Gamut Mapping

camera’s gamut

sRGB gamut

Gamut mapping is necessary because the gamut of the camera’s color space is different

from the gamut of sRGB.

Gamut mapping is a natural mechanism to support picture styles, such as portrait,

landscape, etc.

Color rendering Gamut mapping (color adjustment part)

From Canon webpage

Proposed a new model

Gamut Mapping (h)

Tone Mapping ( f )

RAW

Tw sRGB

Ts

RAW to sRGB (Ts)

White Balance (Tw) sRGB (JPEG)

Camera (RAW)

)(

bx

gx

rx

bx

gx

rx

E

E

E

hf

i

i

i

T

Gamut Mapping (h)

Tone Mapping ( f )

RAW

Tw sRGB

Ts

RAW to sRGB (Ts)

White Balance (Tw) sRGB (JPEG)

Camera (RAW)

Proposed a new model

sRGB Image to RAW

based on several sRGB-RAW image pairs,

• f-1 & T-1 are computed using less saturated points

• h-1 is computed with scatter point interpolation via radial basis func.

bx

gx

rx

e

e

e

h

)(

)(

)(

1

1

1

bx

gx

rx

if

if

if

=

bx

gx

rx

E

E

E

h T=

RAW

sRGB (JPEG)

RAW to sRGB (Ts)

Gamut Mapping (h)

Tone Mapping ( f )

sRGB

Tw

Ts

White Balance (Tw) Camera (RAW)

Canon EOS1Ds Mark III

… … Ts

Tw1

White Balance

Picture Styles

h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3

Standard Portrait Landscape S P L

RAW

sRGB (JPEG)

RAW to sRGB (Ts)

Gamut Mapping (h)

Tone Mapping ( f )

sRGB

Canon EOS1Ds Mark III

… … Ts

Tw1

White Balance

Picture Styles

h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3

Tw

Ts

White Balance (Tw) Camera (RAW)

Standard Portrait Landscape S P L

Gamut Mapping

Mapping is represented as a displacement map of the camera’s original RGB value to its sRGB location.

Experiments : Mapping Image to RAW

Experiments : Mapping Image to RAW

Canon EOS1D

input sRGB image ground truth RAW

Experiments : Mapping Image to RAW

Canon EOS1D

input sRGB image estimated RAW

Experiments : Mapping Image to RAW

Canon EOS1D

new model (f, T, h) old model (f, T)

We cannot handle fully saturated points.

Experiments : Mapping Image to RAW

Canon EOS550D

input sRGB image ground truth RAW

Experiments : Mapping Image to RAW

Canon EOS550D

input sRGB image estimated RAW

Experiments : Mapping Image to RAW

Canon EOS1D

new model (f, T, h) old model (f, T)

Experiments : Mapping Image to RAW

Sony A200

input sRGB image ground truth RAW

Experiments : Mapping Image to RAW

Sony A200

input sRGB image estimated RAW

Experiments : Mapping Image to RAW

Sony A200

new model (f, T, h) old model (f, T)

Application: Photo Refinishing

RAW

sRGB (JPEG)

RAW to sRGB (Ts)

Gamut Mapping (h)

Tone Mapping ( f )

sRGB

Tw

Ts

White Balance (Tw) Camera (RAW)

Canon EOS1Ds Mark III

… … Ts

Tw1

White Balance

Picture Styles

h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3

Standard Portrait Landscape S P L

Canon EOS1Ds Mark III

… … Ts

Tw1

White Balance

Picture Styles

h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3

RAW

sRGB (JPEG)

RAW to sRGB (Ts)

Gamut Mapping (h)

Tone Mapping ( f )

sRGB

Tw

Ts

White Balance (Tw) Camera (RAW)

Standard Portrait Landscape S P L

Canon EOS1Ds Mark III

… … Ts

Tw1

White Balance

Picture Styles

h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3

RAW

RAW to sRGB (Ts)

Gamut Mapping (h)

Tone Mapping ( f )

sRGB

Tw

Ts

White Balance (Tw) sRGB (JPEG)

Camera (RAW)

Standard Portrait Landscape S P L

Canon EOS1Ds Mark III

… … Ts

Tw1

White Balance

Picture Styles

h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3

RAW

RAW to sRGB (Ts)

Gamut Mapping (h)

Tone Mapping ( f )

sRGB

Tw

Ts

White Balance (Tw) sRGB (JPEG)

Camera (RAW)

Standard Portrait Landscape S P L

What if you took a photo with the wrong settings?

Canon EOS1Ds Mark III

… … Ts

Tw1

White Balance

Picture Styles

h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3

RAW

sRGB (JPEG)

RAW to sRGB (Ts)

Gamut Mapping (h)

Tone Mapping ( f )

sRGB

Tw

Ts

White Balance (Tw) Camera (RAW)

Standard Portrait Landscape S P L

Canon EOS1Ds Mark III

… … Ts

Tw1

White Balance

Picture Styles

h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3

sRGB (JPEG)

Tone Mapping ( f -1)

RAW

RAW to sRGB (Ts

-1)

sRGB Tw

-1

Ts -1

White Balance (Tw -1) Gamut Mapping

(h -1)

Camera (RAW)

Standard Portrait Landscape S P L

Canon EOS1Ds Mark III

… … Ts

Tw1

White Balance

Picture Styles

h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3

sRGB (JPEG)

RAW to sRGB (Ts

-1)

Gamut Mapping (h -1)

Tone Mapping ( f -1)

Tw -1

Ts -1

White Balance (Tw -1)

RAW sRGB

Tw

Ts

Camera (RAW)

Standard Portrait Landscape S P L

Input: cloudy WB + landscape style

Result - Canon EOS 1Ds Mark III

Ground truth: fluorescent WB + standard style

Result - Canon EOS 1Ds Mark III

Photoshop result

Result - Canon EOS 1Ds Mark III

Refinished result

Result - Canon EOS 1Ds Mark III

Ground truth: fluorescent WB + standard style

Result - Canon EOS 1Ds Mark III

Input Ground truth

Our refinished result Photoshop

Result - Canon EOS 1Ds Mark III

Result - Canon EOS 1Ds Mark III

Input: tungsten WB + standard style

Result - Canon EOS 1Ds Mark III

Ground truth: daylight WB + standard style

Result - Canon EOS 1Ds Mark III

Photoshop result

Result - Canon EOS 1Ds Mark III

Our refinished result

Result - Canon EOS 1Ds Mark III

Ground truth: daylight WB + standard style

Input Ground truth

Our refinished result Photoshop

Result - Canon EOS 1Ds Mark III

Result – Nikon D200

Input: tungsten WB + standard style

Result – Nikon D200

Ground truth: daylight WB + standard style

Result – Nikon D200

Photoshop result

Result – Nikon D200

Refinished result

Result – Nikon D200

Ground truth: daylight WB + standard style

Input Ground truth

Photo refinish Photoshop

Result – Nikon D200

Result - Sony α200

Input: tungsten WB + standard style

Result - Sony α200

Ground truth: daylight WB + standard style

Result - Sony α200

Photoshop result

Result - Sony α200

Our refinished result

Result - Sony α200

Ground truth: daylight WB + standard style

Input Ground truth

Our Refinished Result Photoshop

Result - Sony α200

Remember these guys?

Conclusion

• Proposed a new camera processing model – Gamut mapping was introduced

– Allowed us to accurately calibrate for scene modes

– Allowed for accurate remapping from sRGB back to RAW

• Facilitated refinishing application – Camera-specific refinishing

– Our result is what the camera would have performed