Embed Size (px)
Citation preview
Scalable High-Quality 3D Scanning
Karthik NarayanPieter AbbeelJitendra MalikAlexei (Alyosha) EfrosMartin Banks
Electrical Engineering and Computer SciencesUniversity of California at Berkeley
Technical Report No. UCB/EECS-2016-220http://www2.eecs.berkeley.edu/Pubs/TechRpts/2016/EECS-2016-220.html
December 23, 2016
Copyright © 2016, by the author(s).All rights reserved.
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires priorspecific permission.
α β
1
13316
α β
(α,β)(α,β)
2
(a) Ortery Photobench, Perspective
(b) Canon and Carmine Unit
(c) Ortery Photobench, Side
3◦
K
G =∑
s∈S
∑
u∈UI(s, u) +
∑
s1,s2∈SE(s1, s2, u)
I E SU
Q(s, ui) uis
I(s, ui) =∑
uj∈U(||Q(s, ui)−Q(s, uj)||− dij)
2
+∑
l∈Ld(Q(s, ui), l)
+ d(Q(s, ui), p)
dij i jL ui p ui
d(Q(s, ui), p)
E(s1, s2, ui) =||R12Q(s2, ui) + t12 −Q(s2, ui)||2
R12 t12s2 s1
I(s, u) =||uT1 Fu2||2,
Fu1u2
K
x y
3◦
5 120
120
120
3 × 3p max{(max p − pmid), (min p − pmid)}
pmid
T = {(R1, t1), . . . , (Rn, tn)}T ′ = {(R′
1, t′1), . . . , (R
′n, t
′n)} Ri
k Ri[3]Ri
u∑n
i=1(u ·Ri[3])2 uRi[3] R′
iRi Ri[3] u
Ri[3] × u t′i ti{t1, · · · , tn}
{t1, · · · , tn}
< 0.1 n = 120
5
3
(b) KinectFusion [22]
(c) Approach in [25] (Poisson reconstruction) (d) Color images of objects in (a)-(c)
(a) Our method
Camera focusImage plane
Visual hullTrue object
Visual cone
(a) Interactive Segmentation
(b) A&H Detergent
(c) VO5 Shampoo
(d) Softsoap Handsoap
600
5
25
5 600
55
{( i, yi)}i yi = 1yi = 0 pixi
k k = 20{ i|yi = 0}
T TT
T
119 5
30%
30%
F (x) = 1 x ∈ R3 0x0 F (x0) = 1
x0 x0
x0
G(x) = 1 x ∈ R3 1− ϵ0 ϵ = 0.1
5
5
(a) Palmolive detergent, KinectFusion mesh
vertices
(b) Red cup, visual hull mesh
vertices
(c) Red cup, KinectFusion mesh
vertices
(i, j)vh kf
raw c a 4 4
vh (i, j)vh[i, j] =∞ ∞
vh (i, j)
(a) Object color images (b) Raw depth maps (c) KinectFusion meshes (d) Soft visual hull meshes (e) Our method
raw kf (i, j) vh (i, j)
vh kf 1vh
vh kf 1vh kf
(a) Pot, with hallucinated points (b) Pot, hallucinated points removed
(d) Cup holder, hallucinated points removed(c) Cup holder, with hallucinated points
1
vh
vh kf
vh
Merged Refined Depth Cloud,
Camera A OnlyMerged Views, All Cameras
Merged Refined Depth
Cloud, Camera B Only
Sample hallucinated point
Visual coneProof of hallucination
Camera B
Camera A
PP D
P D DP
(a) Post-decimation
(b) Pre-decimation
51
x ∈ R3 F (x) = 1 x x
r = 1 x0
1
(a) Almonds Can
(b) Dove Soap Box (c) Pringles Can (d) 3M Spray
3
√3
19 3
1
33
4
3
0.1
C ←A←V H ←KF ←cloud←
c Ca Araw ← c avh← V H c akf ← KF c anr ← rawnc← rawrefined← nr nc
0 ≤ i < rows0 ≤ j < colsvh[i, j] =∞refined[i, j] =∞ ◃
raw[i, j] =∞ kf [i, j] =∞refined[i, j] = vh[i, j] ◃
|vh[i, j]− kf [i, j]| < 1refined[i, j] = vh[i, j] ◃
◃refined[i, j] = {vh[i, j], kf [i, j]}
newcloud← refinedcloud newcloud
cloud
(a) PR, simple
(e) PR, concave
(i) PR, translucent
(b) VH, simple (c) KF, simple (d) Ours, simple
(f) VH, concave (g) KF, concave (h) Ours, concave
(j) VH, translucent (k) KF, translucent (l) Ours, translucent
(m) Color images, simple objects (n) Color images, concave objects (o) Color images, translucent objects
4
original images our method original images our method
PCL’
s vol
umet
ric b
lendi
ng [1
7]
PCL’
s vol
umet
ric b
lendi
ng [1
7]
6003
1
133
MM P p ∈ P
N(p) ⊂ P {Ii}
Ki ∈ R3×3
T0i ∈ SE(3) p ∈ P C(p) p
C = {C(p)}
CT0 = {T0
i }V (p) ⊂ {Ii}
p Ti IiΓi(p,Ti) p Ii Ti
Ki Ii ∈ V (p) ∥C(p)− Γi(p,Ti)∥2
J (C,T) =1
2
∑
p∈P
∑
Ii∈V (p)
∥C(p)− Γi(p,Ti)∥2
T = {Ti} C T Γi(p,Ti)Ti Ki
Ii Γi(ui(g(p,Ti))) g u
g(p,Ti) = Tip
u([gx, gy, gz, gw]T ) = (cx + gxfx/gz, cy + gyfy/gz)
fx, fy Ii (cx, cy) IiKi Γi([ux, uy]T )
(ux, uy) Ii
0 1
(a) Iteration 0 (b) Iteration 200
T C(p)Γi(p,Ti) Ii ∈ V (p)
C(p)
600
V (p) ppIi
p −1V (p)
N p N 1600
NN
N N = 1
NN 30
N NN = 1
N = 10N
p tp ptp = 1 p 0
J (C,T) =1
2
∑
p∈P
∑
Ii∈V ′(p;tp)
∥C(p)− Γi(p,Ti)∥2
V ′(p; tp) N = 30 tp = 1 N = 10N = 10, 30 tp
tp
N = |V ′(p, tp)|
Ii Ii p
(a) N = 1 (b) N = 10 (c) N = 30 (c) N = 50 (d) N = 100 (e) N = 200 (f) N = 600
N = |V ′(p; tp)|N
N = 1
N
M
J (C,T) =1
2
∑
p∈P
∑
Ii∈V ′(p;tp)
∥C(p)− Γi(p,Ti)∥2+
λ
2
∑
p∈P
∑
p′∈N(p)
(1− tptp′) · ∥C(p)− C(p′)∥2
λ
(a) No smoothing, λ = 0 (b) Smoothing, λ = 10
Boundary artifacts
Boundary artifacts
gone
J (C,T)
r(1)i,p = C(p)− Γi(p,Ti)
r(2)p,p′ = C(p)− C(p′)
Ck Tk C T k xk = [Ck,Tk]x0 = [C0,T0] T0
C0(p) {Γi(p,T0i )}Ii∈V ′(p;tp)
xk+1 = xk +∆xk
∆xk
JTJ∆xk = −JT r
r = [r(1), r(2)] J = [Jr(1) , Jr(2) ] rxk
r(1) = [r(1)i,p(x)|x=xk ](i,p)
r(2) = [r(2)i,p(x)|x=xk ](i,p)
Jr(1) = [∇r(1)i,p(x)|x=xk ](i,p)
Jr(2) = [∇r(2)i,p(x)|x=xk ](i,p)
J |P|
100
J (C,T) C TC
x0 = [C0,T0]C J (C,T) C T
∇C(p)J (C;T) =∑
Ii∈V ′(p; tp)
[C(p)− Γi(p,Ti)]+
λ∑
p′∈N(p)
(1− tptp′) · [C(p)− C(p′)]
T J (C,T) T CT
J (C,T) =1
2
∑
Ii
∑
{p:Ii∈V ′(p; tp)}
∥r(1)i,p∥2
IiIi rj,p Tj i = j
Ii xk = [Ck,Tk]Ck J r
r = [r(1)i,p(x)|x=xk ](i,p)
J = [∇r(1)i,p(x)|x=xk ](i,p)
r J
ri,p T r(1)i,p
r(1)i,p Γi(p,Ti) Γi(p,Ti)
Ti Tki ξi = (αi,βi, γi, ai, bi, ci)T
Ti ≈
⎛
⎜⎜⎝
1 −γi βi aiγi 1 −αi bi−βi αi 1 ci0 0 0 1
⎞
⎟⎟⎠Tki
∇Tir(1)i,p = − ∂
∂ξi(Γi(p,Ti)) = −
∂
∂ξi(Γi(ui(g(p,Ti))))
= −∇Γi(u)Ju(g)Jg(ξi)|x=xk
Jg(ξi) Ju(g)∇Γi(u) Γi(u)
u Ii ∆xk
Ti
(a) Level 1, optimized (b) Level 2, optimized (b) Level 3, optimized
√3
ξi SE(3) Tk+1iT |{Ii}|
6
Ti JTJp
Ii JTJ + ηITi η
0 {p|Ii ∈ V ′(p; tp)} IiIi η 0.001
η 1.1 Ii5 Ti
MM
√3
M
MM0,M1,M2 M1
M2√3
(a) Before depth filtering
(b) After depth filtering
(c) Before SLIC smoothing
(d) After SLIC smoothing
tptp
M0 M−1 50% M0√3
M0
M1
CT M0 M2
2 − 3×M2
tpp ∈ P {Ii}
{I(gray)i } I(gray)i10 10 × 10 [−1, 1]
0
I ′i
Ki Ti
n = 133
Ii
9 1 I ′iIi
Ii Ii
I ′i 0 1 tp
tp p p Pi ⊂ PIi
Pi 9Pi p
I ′i p ∈ Pi tpIi
Ii ptp 0 1
64 16 ×4
Original [21] Ours [27] [8] [17] Original [21] Ours [27] [8] [17]
aunt_jemima_original_syrup mom_to_mom_butternut_squash_pear
white_rain_sensations_apple_blossomdetergent
palmolive_green pepto_bismol
cholula_chipotle_hot_saucecoca_cola_glass_bottle
softsoap_whitepop_secret_butter
listerine_greencrest_complete_minty_fresh
crystal_hot_saucewindex
v8_fusion_peach_mango3m_high_tack_spray_adhesive
16
16
1331
0.5
49.1%47.2%
58.1% 47.2%
30% 75%
3
M← P{I(gray)i ,Ki,Ti}← I K
TF ← 10 × 10
[−1, 1] 0I ′ ← {}
I(gray)i ∈ {I(gray)i }I ′i ← Ii
f ∈ FC ← f I(gray)i
I ′i ← I ′i C
Z ← I(gray)i M,Ki,Ti
D ← ZI ′i ← I ′i DI ′i ← (I ′i −min(I ′i))/(max(I ′i)−min(I ′i))S ← Ii
s ∈ Sp← I ′i s
I ′i s p
I ′[i]← I ′i
t← {}p ∈ Pv ← p I ′i pI ′itp ← vtp ← tp ≤ 0.5 1
t
5
m m × m
2 α,βα+β < 1
α < 1α+β > 1
10
20×
α β γ
f γ
α β
D = {x1,x2, · · · ,xm} xi ∈ Rn
E = {y1,y2, · · · ,ym}yi ∈ Rd d = 2 3
Pj|i =exp(−∥xi − xj∥2/2σ2
i )∑ik exp(−∥xi − xk∥2/2σ2
i ),
Pij = (Pi|j +Pj|i)/2m,
Qij =(1 + ∥yi − yj∥2)−1
∑k =l(1 + ∥yk − yl∥2)−1
Pi|i = Qii = 0 σ2i
P·|j k
yi
J (E) =∑
i =j Pij logPij/Qij
∂J∂yi
= 4∑
i =j
Z(yi − yj)(PijQij −Q2ij)
=∑
i =j
4PijQijZ(yi − yj)︸ ︷︷ ︸−∑
i =j
4Q2ijZ(yi − yj)︸ ︷︷ ︸
Z =∑
k =l(1 + ∥yk − yl∥2)−1 O(m2)
Pij
xj xi 3k 0
O(kmn logm) O(km)Pij QijZ = (1+∥yi−yj∥2)−1
yi
O(m logm) yi,yj yk ∥yi−yj∥ ≈ ∥yi−yk∥ ≫ ∥yj−yk∥yj yk
yi
{yi}
yi yi Q2ijZ(yi−yj)
yj
Ncell · Q2ijZ(yi − yj) Ncell
Zzi =
∑j Kq(∥yi − yj∥2) i zi
Z Z
∥yi − ycell∥2/rcell < θ rcell ycellθ
J (E ;α,β) = DαβAB(P∥Q)
1
αβ
∑
i =j
(−Pα
ijQβij +
α
α+ βPα+β
ij +β
α+ βQα+β
ij
),
α ∈ R\{0},β ∈ R β = 0 α+β = 0
Pij Qij
∂J /∂yi
∑
j
4ZQ2ij(yi − yj)(P
αijQ
β−1ij︸ ︷︷ ︸−Qα+β−1
ij − J1 + J2)︸ ︷︷ ︸
J1 =∑
k =l PαklQ
βkl J2 =
∑k =l Q
α+βkl
O(m logm + mk)J1 O(km) P
Z J2
α β
α βλ =
| ln1−α(rij)|
α+ β
∆yi = −∂Dαβ
AB(P∥Q)
∂yi=∑
j
∂Qij
∂yiQλ−1
ij ln1−α
(Pij
Qij
)
=∑
j
∂Qij
∂yiQλ−1
ij ln1−α (rij)
lnq(x) rij = Pij/Qij
fij = ∂Qij/∂yi · Qλ−1ij ln1−α(rij) ∆yi
∂Qij/∂yi k(yi − yj) tk < 0 fij
yj fij yi yj Qλ−1ij ln1−α(rij) > 0 ⇒ rij > 1
yi yj rij < 1D Pij = 0⇒ rij = 0 < 1
70 28× 2832 × 32
100α λ 30
yi 1000
α = 1.0,λ = 1.0
α α λ = 1
lnq(x) ≡ (x1−q − 1)/(1− q) q = 1 lnq(x) = lnx
λ < 1α < 1 λ > 1
1
(α,λ) → (1, 1)
α 1rij < 1 rij > 1
α < 1 rij < 1 ⇒ Qij > Pij yi,yj
fij yi,yj
α < 1α =
λ = 1 α > 14
5 α
α < 1 α > 1α < 1 α > 1 α
λ λ α = 1
λ < 1 fij Qij yi,yj fij Qij yi,yj
λ λ < 1
λ > 1 2λ < 1
λ = 0.95 < 1
3 λ = 1.05 > 1
α βα β
λ = α + β
yi
10−4 yi
200 2500.5 Pij α = 12
750 0.8
yi
yi
PP
13
yi
yi
yi
yi
g1 =∑
j 4ZQ2ij(yi − yj) g2 =
∑j 4ZQ2
ij(yi − yj)Qα+β−1ij J1
J2J2 Z J2 =
∑k =l Q
α+βkl
J (i)2 =
∑k =iQ
α+βki yi J2 =
∑i J
(i)2
J (i)2 yi
ZJ1
yi Pyi 8
(i, j,Pij) 1024i j
(2, 3) (3, 5) y3
g3 =∑
j 4ZQ2ij(yi − yj)Pα
ijQβ−1ij
J1 P9
P PJ1 g3
yi 10 yi
yi g = g1 ∗ (J2 −J1)−g2+g3
10%
yi Z J2J1
yi
θ = 0.25
α = 1,β = 0
20
2 1
503.5
20 − 30
1 2.550
150− 200 55− 60 2< 2 5 − 10
20− 500 20 1− 10100 1
O(logm)
P 1
400%
1000
2 (α = 0.8,λ = 1) (α = 0.95,λ = 0.98)
(α = 0.95,λ = 0.98)
(α = 0.8,λ = 1)
(α = 0.95,λ = 0.98)
28 217
21
0.01
α = 0.98
α = 0.98,λ = 1
J (E) =∑
i,j Pij∥yi − yj∥2 PJ (E) =∑
i,j Pij∥yi − yj∥2 + log∑
k =l exp(−∥xk − xl∥2)
Q
Q 3
L = D − P Dii = (∑
iPij) 0O(km logm + km)
O(m3)
(α,β)
11
Y TLY = I y 0
HIGGS (11M), t-SNE!Time: 1 hour, 1 min
HIGGS (11M), α = 0.98, λ = 1.0!Time: 1 hour, 2 mins
SUSY (5M), α = 0.98, λ = 1.0!Time: 31 mins
Background (Higgs not found)Signal (Higgs found)
Background (Higgs not found)Signal (Higgs found)
Background (SUSY particles not found)Signal (SUSY particles found)
ILSVRC (50K)!α = 0.8, λ = 1.0!Time: 10.4 s
ILSVRC (50K)!t-SNE (α = 1.0, λ = 1.0)!
Time: 10.4 s
ILSVRC (50K)!α = 0.95, λ = 0.98!
Time: 10.4 s
α = 0.8, λ = 1.0, Purple Panel from (a) α = 0.95, λ = 0.98, Red Panel from (c)
α = 0.8, λ = 1.0, Blue Panel from (a) α = 0.95, λ = 0.98, Green Panel from (c)
Entity Implement Covering Container Bird Structure Commodity Invertebrate Hunting Dog Conveyance Mammal
(a) Our method, revealing micro-structures (c) Our method, revealing macro-structures
(h) (i) (j)
Hummingbirds
Toucans
Chickens
Macaws
Swimming Birds
Flamingos
Flying Birds
Monkeys Transitioning to Birds
ChickensBirds
Mammals
Swimming Birds
Bernese Mountain Dogs
Dobermans, Rottweilers
Dalmatians
Black Dogs
Bernese Mountain Dogs
Dobermans, Rottweilers
(b) t-SNE
6
AJ (E ;α,β) =
DαβAB(P∥Q)
1
αβ
∑
i =j
(−Pα
ijQβij +
α
α+ βPα+β
ij +β
α+ βQα+β
ij
),
α ∈ R \ {0},β ∈ Rβ = 0 α+ β = 0
DαβAB(P∥Q) =
∑
ij
d(α,β)AB (Pij ,Qij),
d(α,β)AB (Pij ,Qij) =
⎧⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨
⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩
− 1
αβ
(Pα
ijQβij −
α
α+ βPα+β
ij − α
α+ βQα+β
ij
), α,β,α+ β = 0
1
α2
(Pα
ij lnPα
ij
Qαij
−Pαij +Qα
ij
), α = 0,β = 0
1
α2
(ln
Qαij
Pαij
+Pα
ij
Qαij
− 1
), α = −β = 0
1
β2
(Qβ
ij lnQβ
ij
Pβij
+Pβij
), α = 0,β = 0
1
2(lnPij − lnQij)
2, α = β = 0
α =1,β = 0 α = 1,β = −1 P Q
α = 0
∂D(α,β)AB (P∥Q)/∂yi
α,β = 0
∂D(α,β)AB (P∥Q)
∂yi=
∂
∂yi
[− 1
αβ
(Pα
ijQβij −
α
α+ βPα+β
ij − β
α+ βQα+β
ij
)]
=∂
∂yi
[− 1
αβPα
ijQβij +
1
α(α+ β)Qα+β
ij
]
=∂Qij
∂yi·[− 1
αβPα
ij · β ·Qβ−1ij +
1
α(α+ β)· (α+ β) ·Qα+β−1
ij
]
=∂Qij
∂yi·[− 1
αPα
ij ·Qβ−1ij +
1
α·Qα+β−1
ij
]
= − 1
α· ∂Qij
∂yi·[Pα
ijQβ−1ij −Qα+β−1
ij
]
β
α = 0,β = 0
∂D(α,β)AB (P∥Q)
∂yi=
∂
∂yi
[1
α2
(Pα
ij lnPα
ij
Qαij
−Pαij +Qα
ij
)]
=∂
∂yi
[− 1
α2Pα
ij lnQαij +
1
α2Qα
ij
]
=∂Qij
∂yi
[− 1
α2Pα
ij · αQα−1ij · 1
Qαij
+1
αQα−1
ij
]
=∂Qij
∂yi
[− 1
α
Pαij
Qαij
+1
αQα−1
ij
]
= − 1
α· ∂Qij
∂yi·[Pα
ij
Qαij
−Qα−1ij
]
β = 0
α = −β = 0
∂D(α,β)AB (P∥Q)
∂yi=
∂
∂yi
[1
α2
(ln
Qαij
Pαij
+Pα
ij
Qαij
− 1
)]
=∂
∂yi
[1
α2
(lnQα
ij +Pα
ij
Qαij
)]
=∂Qij
∂yi·[1
α2· αQα−1
ij · 1
Qαij
+1
α2·Pα
ij ·−α · 1
Qα+1ij
]
= − 1
α· ∂Qij
∂yi·[
Pαij
Qα+1ij
− 1
Qij
]
β = −α
Bα β
λλ < 1 λ > 1
α < 1 α > 1
α = 1.0 λ = 1.0 α = 1.2 λ = 0.95 α = 0.8 λ = 1.05
α = 1.0 λ = 1.0 α = 0.9 λ = 0.98 α = 1.05 λ = 0.95
α = 1.0 λ = 1.0 α = 1.2 λ = 1.02 α = 0.6 λ = 1.0
α = 1.0 λ = 1.0 α = 0.8 λ = 1.0 α = 1.0 λ = 0.95
α = 1.0 λ = 1.0 α = 0.6 λ = 1.0 α = 1.0 λ = 0.95
α = 1.0 λ = 1.0 α = 0.6 λ = 1.0 α = 1.0 λ = 0.95
α = 1.0 λ = 1.0 α = 0.8 λ = 1.0 α = 0.95 λ = 0.98
α = 1.0 λ = 1.0 α = 0.6 λ = 0.98 α = 0.95 λ = 0.93
α = 1.0 λ = 1.0 α = 1.0 λ = 0.95
α = 1.0 λ = 1.0 α = 0.6 λ = 1.0
