QualcommApril2013 after · He, J. Sun, and X. Tang, Single image haze removal using dark ... image%haze%removal%using%dark% channel%prior,%Proc.%%IEEE% ... T.%Igarashi,%K.%Nishino,%and%S.K.%Nayar,%The%appearance%of%human%skin.%Technical%Report

RGB++How "Side Information" Improves Computational Photography and Computer Vision(...or how to make better pictures)

Sabine Süsstrunk Images and Visual Representation Group (IVRG)

This%talk%is%about%linking%Informa3on%

Theory%(or%Signal%Processing)%with...

Photography

Source%Coding

Encoder

Decoder

S

R

S

Source%Coding

Encoder

DecoderS

S

Source%Coding

Encoder

DecoderS

S

SystemPerformance

Source%Coding

Encoder

Decoder

Lossless.Source.Coding%S = S

S

S

SystemPerformance

Source%Coding

Encoder

Decoder

Lossy.Source.Coding%S 6= S

S

S

SystemPerformance

Source:%LighAield

Source Encoder

Light!eld

DecoderS

S

SystemPerformance

Encoder:%Camera

Source

Light!eld

Decoder

Camera

S

S

SystemPerformance

Decoder:%Display

Display

Camera

S

S

Light!eld

SystemPerformance

Es3mated%Source:%Image

DisplayImage

Camera

S

S

Light!eld

SystemPerformance

System%Performance:%you%or%me

Camera

DisplayImage

you or me

S

S

Light!eld

Lossless%Source%Coding

DisplayImage

I.like.it!%)

you or me

)

Camera

S

S

Lossy%Source%Coding

)Camera

Display

Light!eld

Image

I.do.NOT.like.it!%

you or me

)

S

S

hEp://www.jigsawexplorer.com/puzzles/disassembledJcameraJjigsawJpuzzle/

Image%Capture

Light!eld Optics

Color Filter Array+

Silicon Sensor

t

Image%Capture

Light!eld Optics

Color Filter Array+

Silicon Sensor

=

t

Source%Coding

DisplayImage

Camera

S

U

Light!eld

S

Remote%Source%Coding

Encoder

Decoder

“Noisy”Observation

U

S

S U

Side%Informa3on

Encoder

Decoder

Side Information

U

S

S “Noisy”Observation

Side%Informa3on

Encoder

Decoder

SideInformation

U

S

S

Y


Side%Informa3on

Encoder

Decoder

SideInformation

U

S

S

Y


Side%Informa3on

Encoder

Decoder

SideInformation

U

S

S

Y

Y


Side%Informa3on

Encoder

Decoder

SideInformation

U

S

S

Y

Y


Side%Informa3on

Encoder

Decoder

SideInformation

U

S

S

Y

Y


Side%Informa3on%can%improve%system%

performance%if%it%is%correlated.with%

the%source%signal.)

Side%Informa3on

Encoder

Decoder

SideInformation“Similar”.images!

U

S

S

Y


Mul3ple%Exposures

Mul3ple%exposures High%dynamic%range%rendering

L.%Meylan,%D.%Alleysson,%and%S.%Süsstrunk,%A%Model%of%Re3nal%Local%Adapta3on%for%the%Tone%Mapping%of%

Color%Filter%Array%Images.%JOSA%A,%2007.

Mul3ple%Illumina3ons

Ambient%Light

Flash Flash/No%Flash%Composite

E.%Eisemann%and%F.%Durand.%Flash%Photography%Enhancement%via%Intrinsic%Religh3ng,%ACM%SIGGRAPH,%2004.

Mul3ple%Time%Instances

Mul3ple%Exposures

“Good”%Face%Composite

M.%F.%Cohen%and%R.%Szeliski.%The%Moment%Camera.%IEEE%Computer,%2006.

Our%Research

Encoder

Decoder

SideInformation

NoisyObservation

Near?Infrared

SemanAcs

U

S

S

Y

Mul$ple'Spectra

400 nm 700 nm 1100 nm

Visible Spectrum Near-Infrared

Visible (Red, Green, and Blue: RGB)

Mul$ple'Spectra

400 nm 700 nm 1100 nm

Visible Spectrum Near-Infrared

Visible (Red, Green, and Blue: RGB) Near-infrared (NIR)

EPFL%

= Ecole Polytechnique Fédéral de Lausanne= Swiss Federal Institute of Technology, Lausanne

Normal%view%(RGB)...

Enhanced%view%(RGB+NIR)

Visible NIR

Haze

sparticle < �/10 : Es �E0

�4

Light.scaBering%in%the%atmosphere%produces%haze%

(Raleigh’s%law):%

Visible NIR

Haze

sparticle < �/10 : Es �E0

�4

Light.scaBering%in%the%atmosphere%produces%haze%

(Raleigh’s%law):%

Intui3on

•The%high.frequencies!of%the%visible%image%can%be%

exchanged!by%the%high%frequencies%of%the%NIR%image.

SpectralJspa3al%correla3on%of%50%RGB/NIR%image%pairs

Z.%Sadeghipoor,%Y.%M.%Lu,%and%S.%Süsstrunk,%Correla3onJbased%Joint%Acquisi3on%and%Demosaicing%of%

Visible%and%NearJinfrared%Images,%ICIP,%2011.

Algorithm

•Mul3Jresolu3on%analysis%and%fusion%of%both%visible%

luminance%V and%NIR%intensi3es%N (criterion:%contrast):%

L.%Schaul,%C.%Fredembach,%and%S.%Süsstrunk,%Color%image%deJhazing%using%the%nearJinfrared,%IEEE%Interna7onal%Conference%on%Image%Processing%(ICIP),%2009.

Idk =

Iak � Ia

k+1

Iak+1

• Analysis%Criterion:

Synthesis

• Keep%the%visible%low%frequency%informa3on,%and%then%

retain%the%highest.contrast%from%either%the%visible%or%

the%NIR%image.%

F0 = V an

�

k=1

(max(V dk , Nd

k ) + 1)


Results

USM Gaussian BilateralVisible

Original Unsharp.Masking

Results


De?hazing.using.NIR

Results

Original


System%Performance?

R. Fa&al, Single image dehazing,

Proc. Interna,onal Conference

on Computer Graphics and

Interac,ve Techniques, 2008.

K. He, J. Sun, and X. Tang, Single

image haze removal using dark

channel prior, Proc. IEEE

Conference on Computer Vision

and Pa>ern Recogni,on, 2009.

System%Performance

R.%FaEal,%Single%image%dehazing,%

Proc.%Interna7onal%Conferenceon%Computer%Graphics%and%Interac7ve%Techniques,%2008.

K.%He,%J.%Sun,%and%X.%Tang,%Single%

image%haze%removal%using%dark%

channel%prior,%Proc.%%IEEE%Conference%on%Computer%Vision%and%PaFern%Recogni7on,%2009.

• NIR%radia3on%has%different%light%scaEering%and%absorp'on/reflec'on%behavior%than%visible%light.

Material%Absorp3on/Reflectances

Yellow%Arrow:%same%color%in%RGB,%different%intensity%in%NIR

Green%Arrow:%different%colors%in%RGB,%same%intensity%in%NIR%

N.%Salama3,%C.%Fredembach,%and%S.%Süsstrunk,%Material%Classifica3on%Using%Color%and%NIR%Images,%IS&T/SID%17th%Color%Imaging%Conference,%2009.

T.%Igarashi,%K.%Nishino,%and%S.K.%Nayar,%The%appearance%of%human%skin.%Technical%Report:%CUCSJ024J05,%2005.

Hair,%fine%wrinkles Epidermis

Dermis

Subcu3s

Hair,%fine%wrinkles

Stratum%corneum

Basal%Cells

Melanosytes%%

Collagenous%network

Blood%vessels

Fat%Cells

Penetra3on%of%radia3on%in%skin

• AbsorpAon%and%scaBering%are%inversely%propor3onal%to%wavelength.

Intui3on

• If%NIR%penetrates%deeper%into%skin,%then%surface%imperfec3ons%are!less%visible.

Intui3on

•Unwanted%skin%imperfec3ons%(freckles,%pores,%warts,%

wrinkles)%are%usually%small.

Base DetailOriginal

NIR

RGB

BF

NIRdetail

NIRbase

Y

Ydetail

BF

Ybase

TRGBJYCC

Ysmooth RGBsmooth

Algorithm

BF=Bilateral%Filter

NIR

RGB

BF

NIRdetail

NIRbase

Y

Ydetail

BF

Ybase

TRGBJYCC

Ysmooth RGBsmooth

Algorithm

BF=Bilateral%Filter

NIR

RGB

BF

NIRdetail

NIRbase

Y

Ydetail

BF

Ybase

TRGBJYCC

Ysmooth RGBsmooth

Algorithm

BF=Bilateral%Filter

Skin%Smoothing

Visible NIR Visible + NIR

C.%Fredembach,%N.%Barbuscia,%and%S.%Süsstrunk,%Combining%visible%and%nearJinfrared%images%for%realis3c%skin%

smoothing,%IS&T/SID%17th%Color%Imaging%Conference,%2009.

Skin%Smoothing




Skin%Smoothing




Skin%Smoothing%



System%Performance?

S.%Süsstrunk,%C.%Fredembach,%and%D.%Tamburrino,%Automa3c%skin%enhancement%with%visible%and%nearJ

infrared%image%fusion,%ACM%Mul7media,%2010.%%Best.Demo.Award.

Camera%Museum%in%Vevey

Visible Visible%+%NIR

S.%Süsstrunk,%C.%Fredembach,%and%D.%Tamburrino,%Automa3c%skin%enhancement%with%visible%and%nearJ

infrared%image%fusion,%ACM%Mul7media,%2010.%%Best.Demo.Award.

Preference:%26.6%%(2,331) Preference:%73.4%%(6,424)

Color%Constancy

Overcast%Sky

Es3mated%Color%Temperature:

7500K

Sunny%Day


5000K

Incandescent%Lamp


3000K

C. Fredembach and S. Süsstrunk, Illuminant estimation and detection using near infrared, IS&T/SPIE Electronic Imaging, Digital Photography V, Vol. 7250, 2009.

Shadow%Detec3on

Visible NIR Binary.Shadow.Mask

D. Rüfenacht, C. Fredembach, and S. Süsstrunk, “Automatic and Accurate Shadow Detection Using Near-infrared Information,” submitted to IEEE Transactions on Pattern Recognition and Machine Intelligence, 2013.

Comparison%with%visible%only

VisibleIntrinsic!Image

Visible!Segmenta7on

VisibleVisible

Intrinsic

NIR

Visible!OnlyVisible!+!NIR

N.%Salama3%and%S.%Süsstrunk,%MaterialJBased%Object%Segmenta3on%using%NearJInfrared%Informa3on,%

IS&T/SID%18th%Color%Imaging%Conference,%2010.%Best.InteracAve.Paper.Award

Image%Database%(477%RGB+NIR)

• 477%image%pairs%for%9%scene%categories:

M.%Brown%and%S.%Süsstrunk,%Mul3Jspectral%SIFT%for%Scene%Category%Recogni3on,%CVPR%2011.

Seman3c%Image%Segmenta3on

N.%Salama3,%D.%Larlus,%G.%Csurka,%and%S.%Süsstrunk,%Seman3c%Image%Segmenta3on%Using%Visible%and%NearJ

Infrared%Channels,%in%ECCV’s%4th%Color%and%Photometry%in%Computer%Vision%Workshop%(2012).

Visible NIR V%Seg V+NIR%Seg GT

Why%NearJInfrared?

Near Infrared Blocking Filter (Hot Mirror)

Camera%Design

All%digital%cameras%are%inherently%

sensi3ve%to%NIR…

All%digital%cameras%are%inherently%

sensi3ve%to%NIR…

...if we remove the hot mirror!

Side%Informa3on

Encoder

Decoder

NIRImages

Y

U

S

S “Noisy”Observation

RGBN%Camera

Encoder

Decoder

Less “Noisy”Observation

with NIR

U

S

S

Seman3cs

Encoder

Decoder

WORDS

U

S

S

Y


Which%image%do%you%like%beEer?

sand

sunset

Which%image%do%you%like%beEer?

dark snow

Image%rendering%depends%on%context

Camera%Modes

“automa3c”%mode “foilage”%mode

Canon%PowerShot%S100

Correla3on?

?

How%do%we%link%image%characteris3cs%with%words?

Wordl%of%the%wikipedia%page%“word”

•MIR%Flickr%database,%1%Million%annotated%images.

• Selec3on%based%on%Flickr’s%“interes3ngness”%score.

• 1%MegaPixel,%assume%sRGB.

gold,%oregoncoast,%fortstevens,%astoria,%outside,%

lightroom,%sigma,%1020mm,%nikon,%d40,%

diamondclassphotographer,%grass,%yellow,%blue,%sky,%

clouds,%singlecloud,%color,%saturated,%happy,%field

Meredith_Farmer (cc)

MJ%Huiskes,%B%Thomee,%MS%Lew.%New%trends%and%ideas%in%visual%concept%detec3on:%the%MIR%Flickr%

retrieval%evalua3on%ini3a3ve,%ACM%Mul7media,%2010.

Image%Database

1M%images%+%keywords

goldgold

996’6883312

Sta3s3cal%Framework

Sta3s3cal%Framework

goldgold

stevewhis (cc) laura.bell (cc)

paige_eliz (cc) Arty Smokes (cc)

TW Collins (cc)raketentim (cc)

golfnride (cc)Dunechaser (cc)

Tal Bright (cc) 10863752@N00 (cc)

4 6

Sta3s3cal%Framework

goldgold

stevewhis (cc) laura.bell (cc)

paige_eliz (cc) Arty Smokes (cc)

TW Collins (cc)raketentim (cc)

golfnride (cc)Dunechaser (cc)

Tal Bright (cc) 10863752@N00 (cc)

percentage%of%yellow%pixels

10%90%

70%5%

8%

0%

30%

3%

2% 9%

MannJWhitneyJWilcoxon%ranksum%test

cardinali3es

of%both%sets

F.%%Wilcoxon,%Individual%comparisons%by%ranking%methods,%Biometrics%Bulle3n,%1(6):80–83,%1945

�2T =

nwnw(nw + nw + 1)

12µT =

nw(nw + nw + 1)

2

nw, nw z =T � µT

�T=

30� 22

4.69⇡ 1.71

sorted%list: 10% 90%70%5% 8%0% 30%3%2% 9%1 2 3 4 5 6 7 8rank%index:

ranksum:

9 10T = 4 + 7 + 9 + 10 = 30

Sta3s3cal%Framework

MannJWhitneyJWilcoxon%ranksum%test

cardinali3es

of%both%sets

�2T =

nwnw(nw + nw + 1)

12µT =

nw(nw + nw + 1)

2

nw, nw z =T � µT

�T=

30� 22

4.69⇡ 1.71

sorted%list: 10% 90%70%5% 8%0% 30%3%2% 9%1 2 3 4 5 6 7 8rank%index:

ranksum:

9 10T = 4 + 7 + 9 + 10 = 30

Sta3s3cal%Framework

significantly%more%yellow%pixels%in%gold%images.z > 0

• CIELAB%histogram

15x15x15%bins.

• %%%values%indicate%significance%of%a%

keyword%w.r.t.%to%a%

characteris3c.

gold

z

z

%%%%Distribu3on

A.%Lindner,%A.%Shaji,%N.%Bonnier,%and%S.%Süsstrunk,%Joint%Sta3s3cal%Analysis%of%Images%and%Keywords%

with%Applica3ons%in%Seman3c%Image%Enhancement,%ACM%Mul7media,%2012.

Other%Characteris3cs

Spa3al%lightness%layout.

light

−7

−6

−5

−4

−3

−2

−1

0

1



Spa3al%chroma%layout.

barn

−6

−4

−2

0

2

4

6

8




Spa3al%Gabor%filter%layout.

!reworks

−10

−5

0

5




Seman3c%Image%Enhancement

Gray%scale%tone%

mapping

snow

goldColor%enhancement%

macroChange%depthJofJfield

[Zhuo%and%Sim,%2011]

Seman3c%Enhancement

seman3c

processing

input

outputimage

component

seman3c

componentgold

characteris3cs

seman3c

component



Seman3c%Component

0 50 100 150 200 250

0

1

2

3

4

5

6

pixel value

z va

lue

redgreenblue

significance%values%for%gold

0 100 2000

50

100

150

200

250

input valueou

tput

val

ue

redgreenblueidentity

global%scale%parameter

f 0 =

⇢1/ (1 + Sz) if z � 01 + S|z| if z < 0

S

Tone%mapping%func3on f

z

Seman3c%Enhancement

seman3c

processing

input

output

gold

characteris3cs

0 100 2000

50

100

150

200

250

input value

outp

ut v

alue


image

component



Image%Component

gold weight%map

Gaussian%blurring%kernel

(1%%of%image%diagonal)

! =

⇥g� ⇤ zw

�col(p)

�⇤10

g�⇥·⇤10normaliza3on%operator

!

Seman3c%Enhancement

input

output

gold

characteris3cs

0 100 2000

50

100

150

200

250

input value

outp

ut v

alue


seman3c

processing



Seman3c%Enhancement

Enhance%relevant%characteris3cs%in%relevant%regions.

input

output

gold

characteris3cs

0 100 2000

50

100

150

200

250

input value

outp

ut v

alue


Iout

= (1� !) · Iin

+ ! · Itmp

Iout

= (1� !) · Iin

+ ! · Itmp

!

Itmp



sand

sand

snow

snow

dark

dark

silhouette

silhouette

sunset

sunset

grass

grass

autumn

autumn

strawberry

strawberry

sky

sky

banana

banana

macro

macro

flower

flower

macro

macro

sand

System%Performance?

%%8%keywords,%30%images,%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%,%30%observers

%%=28’800%image%comparisons.

S = {0.5, 1, 2, 4}

original proposed

Amazon%Mechanical%Turk

0.5 1.0 2.0 4.0

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

S

appr

oval

rate

whitedarksandsnowcontrastsilhouetteportraitlight



Conclusion

• Adding%side.informaAon%to%the%encoding,%decoding%or%

both%can%improve%system.performance,

. and%that%can%be%“like”%or%“dislike”%of%photos.

Conclusion

• Adding%side%informa3on%to%the%encoding,%decoding%or%

both%can%improve%system%performance

. and%that%can%be%like%or%dislike%of%photos• The%given%examples%are%in%image.enhancement%

(computa3onal%photography),%but%the%same%concept%is%

also%applicable%to%computer.vision%applica3ons.

Conclusion

• Considering%todays%capture%devices%(cell%phones,%android%

cameras)%with%mulAple.sensors.and.connected.to.the.

internet,%there%is%more%side%informa3on%available%than%

ever!

hEp://gizmoreport.com/huaweiJascendJmateJinfo/

Conclusion

• Considering%todays%capture%devices%(cell%phones,%android%

cameras)%with%mulAple.sensors.and.connected.to.the.

internet,%there%is%more%side%informa3on%available%than%

ever!

hEp://www.newgadgets.de/46122/nikonJcoolpixJs800cJ16JmegapixelJkameraJmitJandroid/

Conclusion

• Side%Informa3on%needs%to%be%correlated%with%the%signal,%

and%that%might%not%always%be%so%evident%with%nonJimage%

signals.

. Let%your%imagina3on%rule!

Acknowledgment

• Joint work with

J Nathalie Barbuscia, EPFL

J Nicolas Bonnier, OCE now Canon CiSRA

J Clément Fredembach, EPFL now Canon CiSRA

J Albrecht Lindner, EPFL

J Yue M. Lu, EPFL now Harvard

J Neda Salamati, EPFL

J Zahra Sadeghipoor, EPFLJ Lex Schaul, EPFL now ETHZ

J Appu Shaji, EPFL

J Daniel Tamburrino, EPFL now ECAL• Funding

J Swiss National Science Foundation

J OCE Print Logic, a Canon Group

J Hasler Foundation

J Xerox Foundation

Thank%you!

ivrg.epfl.ch

Documents

QualcommApril2013 after · He, J. Sun, and X. Tang, Single image haze removal using dark ... image%haze%removal%using%dark% channel%prior,%Proc.%%IEEE% ... T.%Igarashi,%K.%Nishino,%and%S.K.%Nayar,%The%appearance%of%human%skin.%Technical%Report