A New In-Camera Imaging Model for
Color Computer Vision and its Application
Hai Ting LIN Supervisor: Prof. Brown
National University of Singapore
May 9th, 2012
Joint work with: Seon Joo Kim, Lu Zheng, Sabine Süsstrunk, Stephen Lin
Talk overview
• Motivation – Modern digital cameras
– Cameras and computer vision
• Our work (ICCV’11) – A new imaging pipeline for cameras
• Application of our pipeline – sRGB to RAW
– Photo refinishing (and back)
Camera = light-measuring device
From Digital Image Processing, Gonzales/Woods
Simple models assume an image is a measurement of scene radiance.
Illumination source (Scene radiance)
Internal Image Plane
Scene Element
Imaging System
Output (digital) image
Camera = light-measuring device
•Shape from shading •Intrinsic image •Image Matching •HDR Imaging •Etc . . .
Shape-from-shading
Image Matching
Lu et al, CVPR’10
From Jon Mooser, CGIT Lab, USC From O’Reilly’s digital media forum
HDR Imaging
Light-measuring device?
Onboard Photofinishing “Secret Recipe” of a Camera
Three different cameras with same aperture, exposure, white-balance and picture style, etc. . .
Can image values be treated as physically meaningful values?
And if so, when and how?
Modern photography pipeline
8
In-Camera CCD response (RAW)
CCD Demosaicing (RAW)
“Photo-finishing Processing”
Scene Radiance
Starting point: reality (in radiance)
Camera Output: sRGB
Pre-Camera Lens Filter
Lens Shutter
Aperture
Digital camera pipeline (early work)
Debevec and Malik *SIG’97+
f (x) E*k (RAW)
Unknown f . . . the camera’s non-linear response to RAW.
“Radiometric Calibration”
Accepted model
bx
gx
rx
E
E
E
bx
gx
rx
E
E
E
T
bx
gx
rx
e
e
e
=
)(
)(
)(
bxb
gxg
rxr
ef
ef
ef
bx
gx
rx
i
i
i
=
(RAW) T is a 3x3 matrix i is the sRGB output and f is a non-linear function
Related work
Mann and Picard, SPIE’95 Debevec and Malik, SIG’97 Mitsunaga and Nayar, CVPR’99 Farid, TIP’01 Grossberg and Nayar, TPAMI’03 Grossberg and Nayar, TPAMI’04 Lin et al, CVPR’04 … Manders et al, ICIP’04 Pal et al, CVPR’04 Lin et al, ICCV’05 Kim and Pollefeys, TPAMI’08
Fixe
d p
rop
erty
of
the
cam
era
)(
)(
)(
1
1
1
bxb
gxg
rxr
if
if
if
bx
gx
rx
e
e
e
=
Related work
Mann and Picard, SPIE’95 Debevec and Malik, SIG’97 Mitsunaga and Nayar, CVPR’99 Farid, TIP’01 Grossberg and Nayar, TPAMI’03 Grossberg and Nayar, TPAMI’04 Lin et al, CVPR’04 … Manders et al, ICIP’04 Pal et al, CVPR’04 Lin et al, ICCV’05 Kim and Pollefeys, TPAMI’08 Chakrabarti et al, BMVC’09
Chakrabarti et al conclusions: RAW is meaningful . . . . But, requires a 24 parameter model that is scene-dependent to accurately go back from sRGB to RAW.
Fixe
d p
rop
erty
of
the
cam
era
)(
)(
)(
1
1
1
bxb
gxg
rxr
if
if
if
bx
gx
rx
e
e
e
=
bx
gx
rx
E
E
E
T
bx
gx
rx
e
e
e
=
)(
)(
)(
bx
gxg
rxr
ef
ef
ef
bx
gx
rx
i
i
i
=
Accepted model
Is this model good enough?
Is processing scene dependent?
Observations
Where are the outliers?
Color rendering
Gamut Mapping
camera’s gamut
sRGB gamut
Gamut mapping is necessary because the gamut of the camera’s color space is different
from the gamut of sRGB.
Gamut mapping is a natural mechanism to support picture styles, such as portrait,
landscape, etc.
Color rendering Gamut mapping (color adjustment part)
From Canon webpage
Proposed a new model
Gamut Mapping (h)
Tone Mapping ( f )
RAW
Tw sRGB
Ts
RAW to sRGB (Ts)
White Balance (Tw) sRGB (JPEG)
Camera (RAW)
)(
bx
gx
rx
bx
gx
rx
E
E
E
hf
i
i
i
T
Gamut Mapping (h)
Tone Mapping ( f )
RAW
Tw sRGB
Ts
RAW to sRGB (Ts)
White Balance (Tw) sRGB (JPEG)
Camera (RAW)
Proposed a new model
sRGB Image to RAW
based on several sRGB-RAW image pairs,
• f-1 & T-1 are computed using less saturated points
• h-1 is computed with scatter point interpolation via radial basis func.
bx
gx
rx
e
e
e
h
)(
)(
)(
1
1
1
bx
gx
rx
if
if
if
=
bx
gx
rx
E
E
E
h T=
RAW
sRGB (JPEG)
RAW to sRGB (Ts)
Gamut Mapping (h)
Tone Mapping ( f )
sRGB
Tw
Ts
White Balance (Tw) Camera (RAW)
Canon EOS1Ds Mark III
… … Ts
Tw1
White Balance
Picture Styles
h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3
Standard Portrait Landscape S P L
RAW
sRGB (JPEG)
RAW to sRGB (Ts)
Gamut Mapping (h)
Tone Mapping ( f )
sRGB
Canon EOS1Ds Mark III
… … Ts
Tw1
White Balance
Picture Styles
h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3
Tw
Ts
White Balance (Tw) Camera (RAW)
Standard Portrait Landscape S P L
Gamut Mapping
Mapping is represented as a displacement map of the camera’s original RGB value to its sRGB location.
Experiments : Mapping Image to RAW
Experiments : Mapping Image to RAW
Canon EOS1D
input sRGB image ground truth RAW
Experiments : Mapping Image to RAW
Canon EOS1D
input sRGB image estimated RAW
Experiments : Mapping Image to RAW
Canon EOS1D
new model (f, T, h) old model (f, T)
We cannot handle fully saturated points.
Experiments : Mapping Image to RAW
Canon EOS550D
input sRGB image ground truth RAW
Experiments : Mapping Image to RAW
Canon EOS550D
input sRGB image estimated RAW
Experiments : Mapping Image to RAW
Canon EOS1D
new model (f, T, h) old model (f, T)
Experiments : Mapping Image to RAW
Sony A200
input sRGB image ground truth RAW
Experiments : Mapping Image to RAW
Sony A200
input sRGB image estimated RAW
Experiments : Mapping Image to RAW
Sony A200
new model (f, T, h) old model (f, T)
Application: Photo Refinishing
RAW
sRGB (JPEG)
RAW to sRGB (Ts)
Gamut Mapping (h)
Tone Mapping ( f )
sRGB
Tw
Ts
White Balance (Tw) Camera (RAW)
Canon EOS1Ds Mark III
… … Ts
Tw1
White Balance
Picture Styles
h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3
Standard Portrait Landscape S P L
Canon EOS1Ds Mark III
… … Ts
Tw1
White Balance
Picture Styles
h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3
RAW
sRGB (JPEG)
RAW to sRGB (Ts)
Gamut Mapping (h)
Tone Mapping ( f )
sRGB
Tw
Ts
White Balance (Tw) Camera (RAW)
Standard Portrait Landscape S P L
Canon EOS1Ds Mark III
… … Ts
Tw1
White Balance
Picture Styles
h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3
RAW
RAW to sRGB (Ts)
Gamut Mapping (h)
Tone Mapping ( f )
sRGB
Tw
Ts
White Balance (Tw) sRGB (JPEG)
Camera (RAW)
Standard Portrait Landscape S P L
Canon EOS1Ds Mark III
… … Ts
Tw1
White Balance
Picture Styles
h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3
RAW
RAW to sRGB (Ts)
Gamut Mapping (h)
Tone Mapping ( f )
sRGB
Tw
Ts
White Balance (Tw) sRGB (JPEG)
Camera (RAW)
Standard Portrait Landscape S P L
What if you took a photo with the wrong settings?
Canon EOS1Ds Mark III
… … Ts
Tw1
White Balance
Picture Styles
h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3
RAW
sRGB (JPEG)
RAW to sRGB (Ts)
Gamut Mapping (h)
Tone Mapping ( f )
sRGB
Tw
Ts
White Balance (Tw) Camera (RAW)
Standard Portrait Landscape S P L
Canon EOS1Ds Mark III
… … Ts
Tw1
White Balance
Picture Styles
h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3
sRGB (JPEG)
Tone Mapping ( f -1)
RAW
RAW to sRGB (Ts
-1)
sRGB Tw
-1
Ts -1
White Balance (Tw -1) Gamut Mapping
(h -1)
Camera (RAW)
Standard Portrait Landscape S P L
Canon EOS1Ds Mark III
… … Ts
Tw1
White Balance
Picture Styles
h1, f1 Tw2 Tw3 Tw4 h2, f2 h3, f3
sRGB (JPEG)
RAW to sRGB (Ts
-1)
Gamut Mapping (h -1)
Tone Mapping ( f -1)
Tw -1
Ts -1
White Balance (Tw -1)
RAW sRGB
Tw
Ts
Camera (RAW)
Standard Portrait Landscape S P L
Input: cloudy WB + landscape style
Result - Canon EOS 1Ds Mark III
Ground truth: fluorescent WB + standard style
Result - Canon EOS 1Ds Mark III
Photoshop result
Result - Canon EOS 1Ds Mark III
Refinished result
Result - Canon EOS 1Ds Mark III
Ground truth: fluorescent WB + standard style
Result - Canon EOS 1Ds Mark III
Input Ground truth
Our refinished result Photoshop
Result - Canon EOS 1Ds Mark III
Result - Canon EOS 1Ds Mark III
Input: tungsten WB + standard style
Result - Canon EOS 1Ds Mark III
Ground truth: daylight WB + standard style
Result - Canon EOS 1Ds Mark III
Photoshop result
Result - Canon EOS 1Ds Mark III
Our refinished result
Result - Canon EOS 1Ds Mark III
Ground truth: daylight WB + standard style
Input Ground truth
Our refinished result Photoshop
Result - Canon EOS 1Ds Mark III
Result – Nikon D200
Input: tungsten WB + standard style
Result – Nikon D200
Ground truth: daylight WB + standard style
Result – Nikon D200
Photoshop result
Result – Nikon D200
Refinished result
Result – Nikon D200
Ground truth: daylight WB + standard style
Input Ground truth
Photo refinish Photoshop
Result – Nikon D200
Result - Sony α200
Input: tungsten WB + standard style
Result - Sony α200
Ground truth: daylight WB + standard style
Result - Sony α200
Photoshop result
Result - Sony α200
Our refinished result
Result - Sony α200
Ground truth: daylight WB + standard style
Input Ground truth
Our Refinished Result Photoshop
Result - Sony α200
Remember these guys?
Conclusion
• Proposed a new camera processing model – Gamut mapping was introduced
– Allowed us to accurately calibrate for scene modes
– Allowed for accurate remapping from sRGB back to RAW
• Facilitated refinishing application – Camera-specific refinishing
– Our result is what the camera would have performed