Scalable High-Quality 3D Scanning - EECS at UC Berkeley€¦ · Scalable High-Quality 3D Scanning...

Scalable High-Quality 3D Scanning

Karthik NarayanPieter AbbeelJitendra MalikAlexei (Alyosha) EfrosMartin Banks

Electrical Engineering and Computer SciencesUniversity of California at Berkeley

Technical Report No. UCB/EECS-2016-220http://www2.eecs.berkeley.edu/Pubs/TechRpts/2016/EECS-2016-220.html

December 23, 2016

Copyright © 2016, by the author(s).All rights reserved.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires priorspecific permission.

α β

1

13316

α β

(α,β)(α,β)

2

(a) Ortery Photobench, Perspective

(b) Canon and Carmine Unit

(c) Ortery Photobench, Side

3◦

K

G =∑

s∈S

∑

u∈UI(s, u) +

∑

s1,s2∈SE(s1, s2, u)

I E SU

Q(s, ui) uis

I(s, ui) =∑

uj∈U(||Q(s, ui)−Q(s, uj)||− dij)

2

+∑

l∈Ld(Q(s, ui), l)

+ d(Q(s, ui), p)

dij i jL ui p ui

d(Q(s, ui), p)

E(s1, s2, ui) =||R12Q(s2, ui) + t12 −Q(s2, ui)||2

R12 t12s2 s1

I(s, u) =||uT1 Fu2||2,

Fu1u2

K

x y

3◦

5 120

120

120

3 × 3p max{(max p − pmid), (min p − pmid)}

pmid

T = {(R1, t1), . . . , (Rn, tn)}T ′ = {(R′

1, t′1), . . . , (R

′n, t

′n)} Ri

k Ri[3]Ri

u∑n

i=1(u ·Ri[3])2 uRi[3] R′

iRi Ri[3] u

Ri[3] × u t′i ti{t1, · · · , tn}

{t1, · · · , tn}

< 0.1 n = 120

5

3

(b) KinectFusion [22]

(c) Approach in [25] (Poisson reconstruction) (d) Color images of objects in (a)-(c)

(a) Our method

Camera focusImage plane

Visual hullTrue object

Visual cone

(a) Interactive Segmentation

(b) A&H Detergent

(c) VO5 Shampoo

(d) Softsoap Handsoap

600

5

25

5 600

55

{( i, yi)}i yi = 1yi = 0 pixi

k k = 20{ i|yi = 0}

T TT

T

119 5

30%

30%

F (x) = 1 x ∈ R3 0x0 F (x0) = 1

x0 x0

x0

G(x) = 1 x ∈ R3 1− ϵ0 ϵ = 0.1

5

5

(a) Palmolive detergent, KinectFusion mesh

vertices

(b) Red cup, visual hull mesh

vertices

(c) Red cup, KinectFusion mesh

vertices

(i, j)vh kf

raw c a 4 4

vh (i, j)vh[i, j] =∞ ∞

vh (i, j)

(a) Object color images (b) Raw depth maps (c) KinectFusion meshes (d) Soft visual hull meshes (e) Our method

raw kf (i, j) vh (i, j)

vh kf 1vh

vh kf 1vh kf

(a) Pot, with hallucinated points (b) Pot, hallucinated points removed

(d) Cup holder, hallucinated points removed(c) Cup holder, with hallucinated points

1

vh

vh kf

vh

Merged Refined Depth Cloud,

Camera A OnlyMerged Views, All Cameras

Merged Refined Depth

Cloud, Camera B Only

Sample hallucinated point

Visual coneProof of hallucination

Camera B

Camera A

PP D

P D DP

(a) Post-decimation

(b) Pre-decimation

51

x ∈ R3 F (x) = 1 x x

r = 1 x0

1

(a) Almonds Can

(b) Dove Soap Box (c) Pringles Can (d) 3M Spray

3

√3

19 3

1

33

4

3

0.1

C ←A←V H ←KF ←cloud←

c Ca Araw ← c avh← V H c akf ← KF c anr ← rawnc← rawrefined← nr nc

0 ≤ i < rows0 ≤ j < colsvh[i, j] =∞refined[i, j] =∞ ◃

raw[i, j] =∞ kf [i, j] =∞refined[i, j] = vh[i, j] ◃

|vh[i, j]− kf [i, j]| < 1refined[i, j] = vh[i, j] ◃

◃refined[i, j] = {vh[i, j], kf [i, j]}

newcloud← refinedcloud newcloud

cloud

(a) PR, simple

(e) PR, concave

(i) PR, translucent

(b) VH, simple (c) KF, simple (d) Ours, simple

(f) VH, concave (g) KF, concave (h) Ours, concave

(j) VH, translucent (k) KF, translucent (l) Ours, translucent

(m) Color images, simple objects (n) Color images, concave objects (o) Color images, translucent objects

4

original images our method original images our method

PCL’

s vol

umet

ric b

lendi

ng [1

7]

PCL’

s vol

umet

ric b

lendi

ng [1

7]

6003

1

133

MM P p ∈ P

N(p) ⊂ P {Ii}

Ki ∈ R3×3

T0i ∈ SE(3) p ∈ P C(p) p

C = {C(p)}

CT0 = {T0

i }V (p) ⊂ {Ii}

p Ti IiΓi(p,Ti) p Ii Ti

Ki Ii ∈ V (p) ∥C(p)− Γi(p,Ti)∥2

J (C,T) =1

2

∑

p∈P

∑

Ii∈V (p)

∥C(p)− Γi(p,Ti)∥2

T = {Ti} C T Γi(p,Ti)Ti Ki

Ii Γi(ui(g(p,Ti))) g u

g(p,Ti) = Tip

u([gx, gy, gz, gw]T ) = (cx + gxfx/gz, cy + gyfy/gz)

fx, fy Ii (cx, cy) IiKi Γi([ux, uy]T )

(ux, uy) Ii

0 1

(a) Iteration 0 (b) Iteration 200

T C(p)Γi(p,Ti) Ii ∈ V (p)

C(p)

600

V (p) ppIi

p −1V (p)

N p N 1600

NN

N N = 1

NN 30

N NN = 1

N = 10N

p tp ptp = 1 p 0

J (C,T) =1

2

∑

p∈P

∑

Ii∈V ′(p;tp)

∥C(p)− Γi(p,Ti)∥2

V ′(p; tp) N = 30 tp = 1 N = 10N = 10, 30 tp

tp

N = |V ′(p, tp)|

Ii Ii p

(a) N = 1 (b) N = 10 (c) N = 30 (c) N = 50 (d) N = 100 (e) N = 200 (f) N = 600

N = |V ′(p; tp)|N

N = 1

N

M

J (C,T) =1

2

∑

p∈P

∑

Ii∈V ′(p;tp)

∥C(p)− Γi(p,Ti)∥2+

λ

2

∑

p∈P

∑

p′∈N(p)

(1− tptp′) · ∥C(p)− C(p′)∥2

λ

(a) No smoothing, λ = 0 (b) Smoothing, λ = 10

Boundary artifacts

Boundary artifacts

gone

J (C,T)

r(1)i,p = C(p)− Γi(p,Ti)

r(2)p,p′ = C(p)− C(p′)

Ck Tk C T k xk = [Ck,Tk]x0 = [C0,T0] T0

C0(p) {Γi(p,T0i )}Ii∈V ′(p;tp)

xk+1 = xk +∆xk

∆xk

JTJ∆xk = −JT r

r = [r(1), r(2)] J = [Jr(1) , Jr(2) ] rxk

r(1) = [r(1)i,p(x)|x=xk ](i,p)

r(2) = [r(2)i,p(x)|x=xk ](i,p)

Jr(1) = [∇r(1)i,p(x)|x=xk ](i,p)

Jr(2) = [∇r(2)i,p(x)|x=xk ](i,p)

J |P|

100

J (C,T) C TC

x0 = [C0,T0]C J (C,T) C T

∇C(p)J (C;T) =∑

Ii∈V ′(p; tp)

[C(p)− Γi(p,Ti)]+

λ∑

p′∈N(p)

(1− tptp′) · [C(p)− C(p′)]

T J (C,T) T CT

J (C,T) =1

2

∑

Ii

∑

{p:Ii∈V ′(p; tp)}

∥r(1)i,p∥2

IiIi rj,p Tj i = j

Ii xk = [Ck,Tk]Ck J r

r = [r(1)i,p(x)|x=xk ](i,p)

J = [∇r(1)i,p(x)|x=xk ](i,p)

r J

ri,p T r(1)i,p

r(1)i,p Γi(p,Ti) Γi(p,Ti)

Ti Tki ξi = (αi,βi, γi, ai, bi, ci)T

Ti ≈

⎛

⎜⎜⎝

1 −γi βi aiγi 1 −αi bi−βi αi 1 ci0 0 0 1

⎞

⎟⎟⎠Tki

∇Tir(1)i,p = − ∂

∂ξi(Γi(p,Ti)) = −

∂

∂ξi(Γi(ui(g(p,Ti))))

= −∇Γi(u)Ju(g)Jg(ξi)|x=xk

Jg(ξi) Ju(g)∇Γi(u) Γi(u)

u Ii ∆xk

Ti

(a) Level 1, optimized (b) Level 2, optimized (b) Level 3, optimized

√3

ξi SE(3) Tk+1iT |{Ii}|

6

Ti JTJp

Ii JTJ + ηITi η

0 {p|Ii ∈ V ′(p; tp)} IiIi η 0.001

η 1.1 Ii5 Ti

MM

√3

M

MM0,M1,M2 M1

M2√3

(a) Before depth filtering

(b) After depth filtering

(c) Before SLIC smoothing

(d) After SLIC smoothing

tptp

M0 M−1 50% M0√3

M0

M1

CT M0 M2

2 − 3×M2

tpp ∈ P {Ii}

{I(gray)i } I(gray)i10 10 × 10 [−1, 1]

0

I ′i

Ki Ti

n = 133

Ii

9 1 I ′iIi

Ii Ii

I ′i 0 1 tp

tp p p Pi ⊂ PIi

Pi 9Pi p

I ′i p ∈ Pi tpIi

Ii ptp 0 1

64 16 ×4

Original [21] Ours [27] [8] [17] Original [21] Ours [27] [8] [17]

aunt_jemima_original_syrup mom_to_mom_butternut_squash_pear

white_rain_sensations_apple_blossomdetergent

palmolive_green pepto_bismol

cholula_chipotle_hot_saucecoca_cola_glass_bottle

softsoap_whitepop_secret_butter

listerine_greencrest_complete_minty_fresh

crystal_hot_saucewindex

v8_fusion_peach_mango3m_high_tack_spray_adhesive

16

16

1331

0.5

49.1%47.2%

58.1% 47.2%

30% 75%

3

M← P{I(gray)i ,Ki,Ti}← I K

TF ← 10 × 10

[−1, 1] 0I ′ ← {}

I(gray)i ∈ {I(gray)i }I ′i ← Ii

f ∈ FC ← f I(gray)i

I ′i ← I ′i C

Z ← I(gray)i M,Ki,Ti

D ← ZI ′i ← I ′i DI ′i ← (I ′i −min(I ′i))/(max(I ′i)−min(I ′i))S ← Ii

s ∈ Sp← I ′i s

I ′i s p

I ′[i]← I ′i

t← {}p ∈ Pv ← p I ′i pI ′itp ← vtp ← tp ≤ 0.5 1

t

5

m m × m

2 α,βα+β < 1

α < 1α+β > 1

10

20×

α β γ

f γ

α β

D = {x1,x2, · · · ,xm} xi ∈ Rn

E = {y1,y2, · · · ,ym}yi ∈ Rd d = 2 3

Pj|i =exp(−∥xi − xj∥2/2σ2

i )∑ik exp(−∥xi − xk∥2/2σ2

i ),

Pij = (Pi|j +Pj|i)/2m,

Qij =(1 + ∥yi − yj∥2)−1

∑k =l(1 + ∥yk − yl∥2)−1

Pi|i = Qii = 0 σ2i

P·|j k

yi

J (E) =∑

i =j Pij logPij/Qij

∂J∂yi

= 4∑

i =j

Z(yi − yj)(PijQij −Q2ij)

=∑

i =j

4PijQijZ(yi − yj)︸ ︷︷ ︸−∑

i =j

4Q2ijZ(yi − yj)︸ ︷︷ ︸

Z =∑

k =l(1 + ∥yk − yl∥2)−1 O(m2)

Pij

xj xi 3k 0

O(kmn logm) O(km)Pij QijZ = (1+∥yi−yj∥2)−1

yi

O(m logm) yi,yj yk ∥yi−yj∥ ≈ ∥yi−yk∥ ≫ ∥yj−yk∥yj yk

yi

{yi}

yi yi Q2ijZ(yi−yj)

yj

Ncell · Q2ijZ(yi − yj) Ncell

Zzi =

∑j Kq(∥yi − yj∥2) i zi

Z Z

∥yi − ycell∥2/rcell < θ rcell ycellθ

J (E ;α,β) = DαβAB(P∥Q)

1

αβ

∑

i =j

(−Pα

ijQβij +

α

α+ βPα+β

ij +β

α+ βQα+β

ij

),

α ∈ R\{0},β ∈ R β = 0 α+β = 0

Pij Qij

∂J /∂yi

∑

j

4ZQ2ij(yi − yj)(P

αijQ

β−1ij︸ ︷︷ ︸−Qα+β−1

ij − J1 + J2)︸ ︷︷ ︸

J1 =∑

k =l PαklQ

βkl J2 =

∑k =l Q

α+βkl

O(m logm + mk)J1 O(km) P

Z J2

α β

α βλ =

| ln1−α(rij)|

α+ β

∆yi = −∂Dαβ

AB(P∥Q)

∂yi=∑

j

∂Qij

∂yiQλ−1

ij ln1−α

(Pij

Qij

)

=∑

j

∂Qij

∂yiQλ−1

ij ln1−α (rij)

lnq(x) rij = Pij/Qij

fij = ∂Qij/∂yi · Qλ−1ij ln1−α(rij) ∆yi

∂Qij/∂yi k(yi − yj) tk < 0 fij

yj fij yi yj Qλ−1ij ln1−α(rij) > 0 ⇒ rij > 1

yi yj rij < 1D Pij = 0⇒ rij = 0 < 1

70 28× 2832 × 32

100α λ 30

yi 1000

α = 1.0,λ = 1.0

α α λ = 1

lnq(x) ≡ (x1−q − 1)/(1− q) q = 1 lnq(x) = lnx

λ < 1α < 1 λ > 1

1

(α,λ) → (1, 1)

α 1rij < 1 rij > 1

α < 1 rij < 1 ⇒ Qij > Pij yi,yj

fij yi,yj

α < 1α =

λ = 1 α > 14

5 α

α < 1 α > 1α < 1 α > 1 α

λ λ α = 1

λ < 1 fij Qij yi,yj fij Qij yi,yj

λ λ < 1

λ > 1 2λ < 1

λ = 0.95 < 1

3 λ = 1.05 > 1

α βα β

λ = α + β

yi

10−4 yi

200 2500.5 Pij α = 12

750 0.8

yi

yi

PP

13

yi

yi

yi

yi

g1 =∑

j 4ZQ2ij(yi − yj) g2 =

∑j 4ZQ2

ij(yi − yj)Qα+β−1ij J1

J2J2 Z J2 =

∑k =l Q

α+βkl

J (i)2 =

∑k =iQ

α+βki yi J2 =

∑i J

(i)2

J (i)2 yi

ZJ1

yi Pyi 8

(i, j,Pij) 1024i j

(2, 3) (3, 5) y3

g3 =∑

j 4ZQ2ij(yi − yj)Pα

ijQβ−1ij

J1 P9

P PJ1 g3

yi 10 yi

yi g = g1 ∗ (J2 −J1)−g2+g3

10%

yi Z J2J1

yi

θ = 0.25

α = 1,β = 0

20

2 1

503.5

20 − 30

1 2.550

150− 200 55− 60 2< 2 5 − 10

20− 500 20 1− 10100 1

O(logm)

P 1

400%

1000

2 (α = 0.8,λ = 1) (α = 0.95,λ = 0.98)

(α = 0.95,λ = 0.98)

(α = 0.8,λ = 1)

(α = 0.95,λ = 0.98)

28 217

21

0.01

α = 0.98

α = 0.98,λ = 1

J (E) =∑

i,j Pij∥yi − yj∥2 PJ (E) =∑

i,j Pij∥yi − yj∥2 + log∑

k =l exp(−∥xk − xl∥2)

Q

Q 3

L = D − P Dii = (∑

iPij) 0O(km logm + km)

O(m3)

(α,β)

11

Y TLY = I y 0

HIGGS (11M), t-SNE!Time: 1 hour, 1 min

HIGGS (11M), α = 0.98, λ = 1.0!Time: 1 hour, 2 mins

SUSY (5M), α = 0.98, λ = 1.0!Time: 31 mins

Background (Higgs not found)Signal (Higgs found)

Background (Higgs not found)Signal (Higgs found)

Background (SUSY particles not found)Signal (SUSY particles found)

ILSVRC (50K)!α = 0.8, λ = 1.0!Time: 10.4 s

ILSVRC (50K)!t-SNE (α = 1.0, λ = 1.0)!

Time: 10.4 s

ILSVRC (50K)!α = 0.95, λ = 0.98!

Time: 10.4 s

α = 0.8, λ = 1.0, Purple Panel from (a) α = 0.95, λ = 0.98, Red Panel from (c)

α = 0.8, λ = 1.0, Blue Panel from (a) α = 0.95, λ = 0.98, Green Panel from (c)

Entity Implement Covering Container Bird Structure Commodity Invertebrate Hunting Dog Conveyance Mammal

(a) Our method, revealing micro-structures (c) Our method, revealing macro-structures

(h) (i) (j)

Hummingbirds

Toucans

Chickens

Macaws

Swimming Birds

Flamingos

Flying Birds

Monkeys Transitioning to Birds

ChickensBirds

Mammals

Swimming Birds

Bernese Mountain Dogs

Dobermans, Rottweilers

Dalmatians

Black Dogs

Bernese Mountain Dogs

Dobermans, Rottweilers

(b) t-SNE

6

AJ (E ;α,β) =

DαβAB(P∥Q)

1

αβ

∑

i =j

(−Pα

ijQβij +

α

α+ βPα+β

ij +β

α+ βQα+β

ij

),

α ∈ R \ {0},β ∈ Rβ = 0 α+ β = 0

DαβAB(P∥Q) =

∑

ij

d(α,β)AB (Pij ,Qij),

d(α,β)AB (Pij ,Qij) =

⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

− 1

αβ

(Pα

ijQβij −

α

α+ βPα+β

ij − α

α+ βQα+β

ij

), α,β,α+ β = 0

1

α2

(Pα

ij lnPα

ij

Qαij

−Pαij +Qα

ij

), α = 0,β = 0

1

α2

(ln

Qαij

Pαij

+Pα

ij

Qαij

− 1

), α = −β = 0

1

β2

(Qβ

ij lnQβ

ij

Pβij

+Pβij

), α = 0,β = 0

1

2(lnPij − lnQij)

2, α = β = 0

α =1,β = 0 α = 1,β = −1 P Q

α = 0

∂D(α,β)AB (P∥Q)/∂yi

α,β = 0

∂D(α,β)AB (P∥Q)

∂yi=

∂

∂yi

[− 1

αβ

(Pα

ijQβij −

α

α+ βPα+β

ij − β

α+ βQα+β

ij

)]

=∂

∂yi

[− 1

αβPα

ijQβij +

1

α(α+ β)Qα+β

ij

]

=∂Qij

∂yi·[− 1

αβPα

ij · β ·Qβ−1ij +

1

α(α+ β)· (α+ β) ·Qα+β−1

ij

]

=∂Qij

∂yi·[− 1

αPα

ij ·Qβ−1ij +

1

α·Qα+β−1

ij

]

= − 1

α· ∂Qij

∂yi·[Pα

ijQβ−1ij −Qα+β−1

ij

]

β

α = 0,β = 0

∂D(α,β)AB (P∥Q)

∂yi=

∂

∂yi

[1

α2

(Pα

ij lnPα

ij

Qαij

−Pαij +Qα

ij

)]

=∂

∂yi

[− 1

α2Pα

ij lnQαij +

1

α2Qα

ij

]

=∂Qij

∂yi

[− 1

α2Pα

ij · αQα−1ij · 1

Qαij

+1

αQα−1

ij

]

=∂Qij

∂yi

[− 1

α

Pαij

Qαij

+1

αQα−1

ij

]

= − 1

α· ∂Qij

∂yi·[Pα

ij

Qαij

−Qα−1ij

]

β = 0

α = −β = 0

∂D(α,β)AB (P∥Q)

∂yi=

∂

∂yi

[1

α2

(ln

Qαij

Pαij

+Pα

ij

Qαij

− 1

)]

=∂

∂yi

[1

α2

(lnQα

ij +Pα

ij

Qαij

)]

=∂Qij

∂yi·[1

α2· αQα−1

ij · 1

Qαij

+1

α2·Pα

ij ·−α · 1

Qα+1ij

]

= − 1

α· ∂Qij

∂yi·[

Pαij

Qα+1ij

− 1

Qij

]

β = −α

Bα β

λλ < 1 λ > 1

α < 1 α > 1

α = 1.0 λ = 1.0 α = 1.2 λ = 0.95 α = 0.8 λ = 1.05

α = 1.0 λ = 1.0 α = 0.9 λ = 0.98 α = 1.05 λ = 0.95

α = 1.0 λ = 1.0 α = 1.2 λ = 1.02 α = 0.6 λ = 1.0

α = 1.0 λ = 1.0 α = 0.8 λ = 1.0 α = 1.0 λ = 0.95

α = 1.0 λ = 1.0 α = 0.6 λ = 1.0 α = 1.0 λ = 0.95

α = 1.0 λ = 1.0 α = 0.6 λ = 1.0 α = 1.0 λ = 0.95

α = 1.0 λ = 1.0 α = 0.8 λ = 1.0 α = 0.95 λ = 0.98

α = 1.0 λ = 1.0 α = 0.6 λ = 0.98 α = 0.95 λ = 0.93

α = 1.0 λ = 1.0 α = 1.0 λ = 0.95

α = 1.0 λ = 1.0 α = 0.6 λ = 1.0