Embed Size (px)
Citation preview
DIG
ITA
LIn
stitu
te o
f Inf
orm
atio
n an
d C
omm
unic
atio
nT
echn
olog
ies
Com
pute
r V
isio
n A
lgor
ithm
s fo
r A
utom
atin
g H
D P
ost-
Pro
duct
ion
Han
nes
Fas
sold
, Jak
ub R
osne
r20
10-0
9-22
2
Ove
rvie
w�
Har
ris/K
LT fe
atur
e po
int d
etec
tion
�K
LT fe
atur
epo
int t
rack
ing
�A
pplic
atio
n�
Rea
l-tim
eH
D s
tabi
lizat
ion
�Im
age
war
ping
�Im
age
inpa
intin
g�
App
licat
ion
�R
e-T
imin
g (‚T
ime-
Str
etch
ing‘
)
�R
esto
ratio
nof
dam
aged
/ mis
sing
fram
es
3
Fea
ture
poi
nt d
etec
tion
Intr
oduc
tion
�F
ind
‘relia
ble’
feat
ure
poin
ts in
imag
e�
Usa
ge�
Cam
era
calib
ratio
n�
Tra
ckin
g�
…
�R
elia
ble
feat
ure
poin
ts h
ave
suffi
cien
t str
uctu
re
in th
eir
loca
l nei
ghbo
rhoo
d�
E.g
. poi
nt w
ithin
hom
ogen
eous
area
notr
elia
ble
�R
elia
ble
feat
ure
poin
tsty
pica
llylo
okco
rner
-like
4
Fea
ture
poi
nt d
etec
tion
Mea
sure
s�
Str
uctu
rem
atrix
G�
2 x
2 m
atrix
Gen
code
sst
ruct
ure
info
rmat
ion
for
an r
ecta
ngul
arar
eaW
(p)
arou
nda
poin
t p
�G
radi
ent i
mag
e →
Cen
tral
Diff
eren
ce, S
obel
orS
harr
oper
ator
�C
orne
rnes
sm
easu
res
�H
arri
sm
easu
re: λ
= d
et(G
) –
k*
trac
e(G
)^2
�K
LT
mea
sure
: λ=
min
imum
eige
nval
ueof
G
�λ
smal
lor
zero
→ho
mog
eneo
usim
age
area
�λ
big
→co
rner
, ric
hly
text
ured
area
5
Fea
ture
poi
ntde
tect
ion
Alg
orith
m�
As
in O
penC
Vro
utin
e‚c
vGoo
dFea
ture
sToT
rack
‘�
Alg
orith
mst
eps
1.C
alcu
late
corn
erne
ssλ
for
all p
ixel
s2.
Cal
cula
tem
axim
umco
rner
nessλ
max
in im
age
3.D
isca
rdal
l pix
els
whi
chλ
smal
ler
than
a fr
actio
nof
λm
ax(e
.g. <
5%
)
4.N
on-m
axim
asu
ppre
ssio
n(d
isca
rd‚w
eak‘
loca
lmax
ima)
5.M
inim
um d
ista
nce
enfo
rcem
ent
�M
inim
um d
ista
nce
enfo
rcem
ent
�E
nsur
esth
atev
ery
feat
ure
poin
t has
a c
erta
inm
inim
umdi
stan
ce to
all
othe
rpo
ints
�A
void
scl
umpi
ngof
mos
tfea
ture
poin
tsin
ric
hly
text
ured
imag
e re
gion
s
�S
ome
issu
esw
ithit
whi
chfo
rce
usto
do
iton
CP
U (
mor
ela
ter)
…
GP
U
CP
U
6
Fea
ture
poi
nt d
etec
tion
Ste
ps 1
-4, C
UD
A im
plem
enta
tion
�A
ll ke
rnel
sm
ake
exte
nsiv
e us
age
of s
hare
dm
emor
y�
Cor
nern
ess
calc
ulat
ion
�T
hree
kern
els
for
conv
olut
ion,
str
uctu
rem
atrix
, cor
nern
ess
(KLT
form
ula)
�D
eter
min
em
axim
umco
rner
nessλ
max
�Is
a re
duct
ion
oper
atio
n→
CU
DP
P li
brar
y
�D
isca
rdfe
atur
epo
ints
with
low
corn
erne
ss�
Set
a ‚d
isca
rdfla
g‘fo
rea
chpi
xelt
o be
disc
arde
d
�N
on-m
axim
asu
ppre
ssio
n�
Ker
neli
sva
riatio
nof
dila
teop
erat
or
�(B
efor
e5)
Tra
nsfe
r no
n-di
scar
ded
pixe
lsto
CP
U�
Bef
ore
tran
sfer
, all
disc
arde
dpi
xels
are
filte
red
out b
ya
com
pact
ion
oper
atio
n→
CU
DP
P li
brar
y
7
Fea
ture
poi
nt d
etec
tion
Ste
p 5
�M
inim
um d
ista
nce
enfo
rcem
ent
�G
iven
: ‚ca
ndid
ate
list‘
(all
pixe
lsw
hich
have
n‘tb
een
disc
arde
d)�
Itera
teth
roug
hlis
t, st
artin
gw
ithca
ndid
ates
with
high
estc
orne
rnes
s, a
nd
add
them
to o
utpu
tlis
t�
Bef
ore
addi
nga
cand
idat
e, it
sdi
stan
ce to
all
poin
tsal
read
yin
out
putl
ist
isch
ecke
d
�Is
sues
�P
roce
ssis
inhe
rent
lyse
rial→
forc
esus
to d
o on
CP
U�
Ope
nCV
impl
emen
tatio
nno
teffi
cien
tfor
seve
ralt
hous
and
poin
ts�
Dev
elop
edal
tern
ativ
e m
etho
d�
Prin
cipl
e: W
hen
addi
nga
cand
idat
e, t
heci
rcul
arar
eaar
ound
itis
mar
ked
as ‚o
ccup
ied‘
.�
Line
ar c
ompl
exity
�A
utom
atic
sw
itchi
ngbe
twee
nO
penC
Van
d al
tern
ativ
e m
etho
d
8
Fea
ture
poi
ntde
tect
ion
Res
ults
9
Fea
ture
poi
ntde
tect
ion
Res
ults
10
Fea
ture
poi
nt d
etec
tion
Run
time
com
paris
on�
GP
U im
pl.:
CU
DA
, GT
X 2
80
�C
PU
impl
.: O
penC
V(u
sing
IPP
), 2
.4 G
hzX
eon
Qua
d-C
ore
�W
indo
wsi
ze=
5 x
5, M
axim
um #
of f
eatu
res
= 1
0000
Run
time
for
the
step
s1
–4
(fea
ture
poin
t det
ectio
nw
ithou
tmin
imum
dist
ance
enf
orce
men
t)R
untim
efo
rst
ep5
(min
imum
dist
ance
enf
orce
men
t)
11
Fea
ture
poi
nt tr
acki
ngIn
trod
uctio
n�
Fea
ture
poi
nt tr
acki
ng�
Giv
ena
spar
sese
tof f
eatu
repo
ints
in c
urre
ntim
age
I(e.
g. fo
und
byfe
atur
epo
int d
etec
tion)
, fin
d th
eir
posi
tion
in s
ubse
quen
tim
age
J
�Im
port
antl
ow-le
velt
ask
in c
ompu
ter
visi
on�
Use
dfo
rob
ject
trac
king
, cam
era
mot
ion
estim
atio
n,
stru
ctur
efr
omm
otio
n, …
�K
LT a
lgor
ithm
(Kan
ade,
Luc
as, T
omas
i)�
Ver
ypo
pula
rm
etho
d�
Rea
sona
bly
fast
, ful
lyau
tom
atic
, suf
ficie
ntqu
ality
�F
or e
ach
fram
eI i
n se
quen
ce�
Det
ectn
ewfe
atur
esin
Ian
d ad
dth
emto
alre
ady
exis
ting
ones
�T
rack
all
feat
ures
from
I to
subs
eque
ntim
age
J
12
Fea
ture
poi
nt tr
acki
ngA
lgor
ithm
prin
cipl
e�
Dis
sim
ilarit
yfu
nctio
n�
p…
poin
t, v
.. m
otio
nve
ctor
,W
(p)
.. n
x n
win
dow
cent
ered
at p
�F
or e
ach
poin
t p, f
ind
mot
ion
vth
atm
inim
izesε(v
)�
Min
imiz
atio
nof
ε(v
)�
Gra
dien
t des
cent
met
hod
(iter
ativ
e m
etho
d, G
auss
-New
ton
type
)�
Gra
dien
t des
cent
met
hods
need
‚goo
d‘in
itial
valu
ev 0
�C
reat
em
ulti-
reso
lutio
nim
age
pyra
mid
�D
o m
inim
izat
ion
on e
ach
leve
lof p
yram
id
�S
olut
ion
of le
velm
+ 1
isus
edas
initi
aliz
atio
nfo
rle
velm
13
Fea
ture
poi
nt tr
acki
ngA
lgor
ithm
�P
seud
o-C
ode
�F
or o
neP
yram
id L
evel
, for
one
poin
t�
Typ
ical
ly: W
(p)
= 5
x 5
pix
el, m
axIte
r=
10,
eps
= 0
.03
14
Fea
ture
poi
nt tr
acki
ngC
UD
A im
plem
enta
tion
�G
auss
ian
imag
e py
ram
id�
Con
volu
tion
+ s
ubsa
mpl
ing
�F
eatu
re p
oint
trac
king
(key
issu
es)
�O
ne k
erne
lcal
lfor
each
pyra
mid
leve
l
�O
ne th
read
= o
nepo
int
�G
PU
und
er-u
tiliz
atio
nif
# po
ints
isto
osm
all(
e.g.
som
ehu
ndre
dpo
ints
)
�R
educ
e#
of te
xtur
efe
tche
s, e
spec
ially
in in
ner
loop
�E
ach
thre
adne
eds
lot o
f sha
red
mem
ory
�E
spec
ially
for
bigg
erw
indo
wsi
zes
(9x9
, 11x
11, .
.) th
isle
ads
tolo
wm
ultip
roce
ssor
occu
panc
y
�D
iffic
ultt
o fin
d be
st c
ompr
omis
e(t
hrea
dbl
ock
size
, # r
egis
ters
per
thre
ads…
)�
Lot o
f exp
erim
enta
tion
nece
ssar
y, n
eed
to im
plem
entd
iffer
ent v
aria
nts
15
Fea
ture
poi
nt tr
acki
ngR
esul
ts
16
Fea
ture
poi
nt tr
acki
ng
Run
time
com
paris
on�
GP
U im
pl.:
CU
DA
, GT
X 2
80
�C
PU
impl
.: O
penC
V(u
sing
IPP
), 2
.4 G
hzX
eon
Qua
d-C
ore
�F
ullH
D(1
920
x 10
80),
win
dow
size
= 5
x 5
, #le
vels
= 6
, max
Iter
= 1
0, e
ps=
0.0
3
17
Fea
ture
poi
nt tr
acki
ngA
pplic
atio
n: S
tabi
lizat
ion
�P
robl
em�
Ann
oyin
gfil
m e
xper
ienc
edu
eto
imag
e ‚v
ibra
tion‘
�P
ossi
ble
reas
ons
�sh
aky
cam
era
�in
stab
ility
in fi
lm tr
ansp
ortd
urin
gfil
m s
cann
ing
(‚wor
nou
t per
fora
tions
‘)
�W
hatw
ew
ant
�R
educ
e/re
mov
eth
ese
vibr
atio
ns
�…
butl
eave
inte
nded
(typ
ical
ly‚s
moo
th‘)
cam
era
mot
ion
inta
ct
18
Fea
ture
poi
nt tr
acki
ngA
pplic
atio
n: R
ealti
me
HD
Sta
biliz
atio
n�
Alg
orith
mou
tline
�T
rack
feat
ure
poin
tsth
roug
hout
sequ
ence
�R
obus
tlyes
timat
e‚g
loba
l‘m
otio
nbe
twee
nco
nsec
utiv
efr
ames
�2-
para
met
er tr
ansl
atio
nalm
odel
(dx,
dy)
�H
ighe
r-pa
ram
eter
mod
elpo
ssib
le(e
.g. a
ffine
mod
el)
�F
ilter
sig
nalt
o ge
tam
ount
of c
orre
ctio
n
�W
arp
fram
esw
ithco
rrec
tion
10 0
-10
19
Fea
ture
poi
nt tr
acki
ngS
tabi
lizat
ion
Dem
o�
Sta
biliz
atio
nw
ith2-
para
met
er tr
ansl
atio
nalm
odel
�W
orks
rea
ltim
e(>
25
fps)
for
Ful
l HD
res
olut
ion
�G
PU
: GT
X 2
85, C
PU
: Qua
dCor
eX
eon
�[V
ideo
_Ste
yrer
_gas
se]
20
Imag
e w
arpi
ngIn
trod
uctio
n�
Giv
enan
sou
rce
imag
e Ia
nd a
non
linea
rm
appi
ngM
, ca
lcul
ate
the
map
ped
imag
e M
(I)
�Im
age
war
ping
exam
ples
�R
otat
ion,
Sca
ling,
..
�A
rbitr
ary
mes
hde
form
atio
ns
�M
appi
ngfu
nctio
nM
�T
ypic
ally
defin
edpi
xel-w
ise
21
Imag
e w
arpi
ngA
lgor
ithm
�U
seac
cum
ulat
orim
age
Aan
d w
eigh
tim
age
W�
float
ing-
poin
tor
fixed
-poi
nt
�A
lgor
ithm
�F
or e
ach
sour
cepi
xelp
�D
eter
min
ede
stin
atio
nlo
catio
nds
t= M
(p)
�In
crem
entt
hefo
ursu
rrou
ndin
gpi
xels
in a
ccum
ulat
oran
dw
eigh
tim
age
→‚b
iline
arw
ritin
g‘
�W
arpe
dim
age
M(I
) =
�P
ixel
-wis
edi
visi
on
�P
ixel
s w
ithw
eigh
tzer
oar
em
arke
das
‚hol
es‘
(no
sour
cepi
xelm
appe
dto
them
)
22
Imag
e w
arpi
ngC
UD
A im
plem
enta
tion
�Is
sues
�A
tom
icop
erat
ions
nece
ssar
yfo
rre
solv
ing
read
-writ
eha
zard
s→
sign
ifica
ntpe
rfor
man
cepe
nalty
for
pre-
Fer
mih
ardw
are
�R
educ
epe
rfor
man
cepe
nalty
�D
eter
min
ea
targ
etre
gion
whe
rem
osto
f thr
eads
of th
ecu
rren
tth
read
bloc
k w
ill li
kely
map
to�
Ass
ume
som
eso
rtof
sm
ooth
ness
in m
appi
ngfu
nctio
nM
�T
arge
t reg
ion
isca
ched
in s
hare
dm
emor
y
�T
hrea
dm
aps
into
targ
etre
gion
→do
sha
red
mem
ory
atom
icop
erat
ion
�T
hrea
ddo
esn‘
tmap
into
targ
etre
gion
→do
glo
balm
emor
yat
omic
oper
atio
n(s
low
er)
23
Imag
e w
arpi
ngS
ourc
e im
age
24
Imag
e w
arpi
ngW
arpe
d im
age
25
Imag
e in
pain
ting
Intr
oduc
tion
�Im
age
inpa
intin
g�
Fill
up u
ndef
ined
regi
ons
in a
n im
age
in th
ebe
st w
ay�
Lot o
f lite
ratu
reab
outi
npai
ntin
gal
gorit
hms
�P
ropa
gate
stru
ctur
e&
text
ure
clev
er in
toho
le
�S
till a
har
dta
sk
�G
oal
�D
evel
opsi
mpl
e an
d fa
st in
pain
ting
algo
rithm
�G
ood
para
lliza
ble
�S
uita
ble
for
hole
soc
curin
gin
war
ped
imag
es�
Thi
n, c
rack
-like
appe
aran
ce
26
Imag
e in
pain
ting
Alg
orith
m�
App
roac
h�
Use
sac
cum
ulat
orim
age
Aan
d w
eigh
tim
age
W�
Det
erm
ine
seto
f hol
e bo
rder
pixe
ls
�F
or e
ach
bord
erpi
xel
�P
ropa
gate
itsin
tens
ityin
toth
eho
le
alon
ga
fixed
seto
f dire
ctio
ns(e
.g. 1
6)
�B
orde
rpi
xeli
nten
sity
prop
agat
ion
�T
race
the
line
from
bord
erpi
xeli
nto
hole
inte
rior
(Bre
senh
am)
�F
or e
ach
visi
ted
pixe
lpits
valu
ein
Aan
d W
isup
date
d
�In
pain
ted
imag
e I h
olef
illed
=
bcu
rr
gd
pA
pA
1)
()
(+
=cu
rrd
pW
pW
1)
()
(+
=
Bor
der
pixe
lpro
paga
tion
alon
gse
vera
ldire
ctio
ns
27
Imag
e in
pain
ting
CU
DA
impl
emen
tatio
n�
Key
issu
es�
One
thre
ad=
one
bord
erpi
xel
�O
ne k
erne
lcal
lper
dire
ctio
n�
All
thre
ads
trac
ein
tosa
me
dire
ctio
ns
�A
tom
icop
erat
ions
no
tus
ed�
Spe
edre
ason
s
�F
loat
ato
mic
oper
atio
nsno
tsup
port
edfo
rpr
e-F
erm
iGP
Us
�R
/W H
azar
dsca
nno
tocc
urfo
r8
mai
ndi
rect
ions
�R
/W H
azar
dsca
noc
cur
(ver
yse
ldom
ly)
for
8 se
cond
ary
dire
ctio
ns→
Indu
ces
negl
igib
ledi
ffere
nces
betw
een
CP
U &
GP
U in
pain
ting
resu
lt
�W
arp
dive
rgen
ce�
Due
to d
iffer
ent p
aths
the
thre
ads
of a
war
par
etr
acin
g
�P
ossi
ble
impr
ovem
ent:
Gro
up th
read
id‘s
byso
me
spat
ialr
elat
ions
hip
28
Imag
e w
arpi
ng &
inpa
intin
gR
untim
e co
mpa
rison
�G
PU
impl
.: C
UD
A, G
TX
285
�C
PU
impl
.: O
wn
optim
ized
impl
., o
ne
CP
U-t
hrea
d(b
utus
esm
ulti-
thre
aded
IPP
-fun
ctio
ns),
Inte
l Xeo
nQ
uad-
Cor
e3.
0 G
hz
�A
vera
geru
ntim
eov
erse
quen
ce, w
arpi
ngfu
nctio
ns=
mot
ion
field
s,
~ 1
.3 %
of w
arpe
dim
ages
to b
ein
pain
ted
29
Imag
e in
pain
ting
Res
ults
30
Imag
e in
pain
ting
& w
arpi
ngA
pplic
atio
n: T
ime-
Str
etch
ing
�T
ime-
Str
etch
ing
effe
ct�
Inse
rtsy
nthe
tical
lyge
nera
ted
fram
esin
vid
eose
quen
ceto
ach
ieve
slow
-m
otio
nef
fect
�G
ener
ate
synt
hetic
fram
ebe
twee
nim
age
I 1an
d I 2
�C
alcu
late
pixe
l-wis
em
otio
n(o
ptic
alflo
w)
betw
een
I 1an
d I 2
�F
ast G
PU
met
hods
avai
labl
e
�S
cale
mot
ion
acco
rdin
gto
des
ired
timep
oint
�W
arp
I 1w
ithsc
aled
mot
ion
�F
illho
les
in w
arpe
dim
age
31
Imag
e in
pain
ting
& w
arpi
ngD
emo:
Tim
e-S
tret
chin
g�
Str
etch
ing
Fac
tor
2.0
�[D
emoV
ideo
‚TU
Mun
ich
pede
stria
nar
ea]
32
Imag
e in
pain
ting
& w
arpi
ngA
pplic
atio
n: R
esto
re d
amag
ed fr
ames
�U
sene
ighb
orfr
ames
to g
ener
ate
‚repl
acem
ent‘
for
dam
aged
fram
e
33
Ack
now
ledg
men
ts�
Sile
sian
Uni
vers
ity, P
olan
d�
Jaku
b R
osne
r
�JO
AN
NE
UM
RE
SE
AR
CH
, Aus
tria
�F
loria
n P
utz,
Her
man
n F
uern
trat
t, W
erne
r B
aile
r, P
eter
S
chal
laue
r, G
eorg
Tha
lling
er
�W
ork
was
sup
port
ed b
y E
urop
ean
Uni
on
proj
ects
�20
20 3
D M
edia
(ht
tp://
ww
w.2
0203
dmed
ia.e
u)
�F
asci
natE
(http
://w
ww
.fasc
inat
e-pr
ojec
t.com
)
�P
rest
oPR
IME
(http
://w
ww
.pre
stop
rime.
eu)
34
DIG
ITA
L-
Inst
itute
of I
nfor
mat
ion
and
Com
mun
icat
ion
Tec
hnol
ogie
s
JOA
NN
EU
M R
ES
EA
RC
H F
orsc
hung
sges
ells
chaf
tmbH
Ste
yrer
gass
e17
, A-8
010
Gra
z, A
US
TR
IA
E-m
ail:
hann
es.fa
ssol
d@jo
anne
um.a
tW
eb:
http
://w
ww
.joan
neum
.at/d
igita
l
Han
nes
Fas
sold
Con
tact
