2D Scene Representation and Description · Level-Set Based Segmentation James Sethian (1999): Level Set & Fast Marching Methods, Cambridge. Stan Osher & Ronald Fedkiw (2002): Level

Institute of Electrical Measurement and Measurement Signal Processing

1

Axel Pinz WS 2017/18 Image and Video Understanding 4

2D Scene Representation and Description

You can get very far in 2D !

2D “image object”

“token”

“tokenset”

2D scene description

image

image

description

segmentation

2D grouping


2


Segmentation: From Images to Tokens

• “Formal” definition of segmentation:Image I is segmented segmentation S = {regions Ri | rules 1-4}

1. ∪ 𝑅𝑖 = 𝐼

2. 𝑖 ≠ 𝑗: 𝑅𝑖 ∩ 𝑅𝑗 = ∅

3. some homogeneity criterion H holds: ∀𝑖: 𝐻 𝑅𝑖 = 𝑡𝑟𝑢𝑒

4. disjunct neighbors: 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑅𝑖 , 𝑅𝑗 : 𝐻 𝑅𝑖 ∪ 𝑅𝑗 = 𝑓𝑎𝑙𝑠𝑒

• Fundamental concepts of segmentation• Region-based segmentation (e.g. threshold, split+merge, R growing)

• Contour-based segmentation (e.g. edge detection closing of gaps)

• State-of-the-art segmentation algorithms• Graph based segmentation

• Level set

image

image

description

segmentation

2D grouping


3


Segmenting 2D “Shapes”

Region-Based Segmentation (1)

e.g.:

Global threshold


4



e.g.: Region growing Seed cells


5



e.g.: Split and Merge

Image


6


Edge-Based Segmentation (1)

General approach:

1) Smoothing

2) Edge detection

3) Filteringa) Eliminate short edges

b) Close small gaps

4) Obtain closed contours

Image Edge image


7



Real edge profile Real line profile

Edge: inflection point

Line: local extremum

Locate it:

Zero crossing of 1st / 2nd derivative !!

Noise smoothing needed convolution with a Gaussian!

Edge

Line


8



D. Marr + E. Hildreth: LoG / DoG zero crossings [1978]

“Mexican hat” / “Sombrero” Operator

http://laurent-duval.blogspot.co.at/2014/09/cours-radial-basis-functions.html

∆ 𝐺 ∗ 𝐼 = (∆𝐺) ∗ 𝐼


9



LoG / DoG zero crossings

always produce closed contours!

Original images

T … threshold

T … threshold

Canny edge

Detector


10


Relaxing the formal definition: Superpixels

Compactness and/or size parameters control the segmentation

Homogeneity may be violated:

disjunct neighbors: 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑅𝑖 , 𝑅𝑗 : 𝐻 𝑅𝑖 ∪ 𝑅𝑗 = 𝑓𝑎𝑙𝑠𝑒… may be true!

Example: [Achanta et al., SLIC superpixels, PAMI 34(11):2274-2282, 2012]


11


SLIC Examples [Achanta et al. PAMI ]

Spatial segmentations

into “superpixels”


12


SLIC Examples [Achanta et al. PAMI ]

Spatio-temporal

segmentations

into “supervoxels”


13


Temporal Superpixels TSPs [Chang et al CVPR‘13]


14


High-Level Segmentation – The “Correct” one?

3 Regions24 Regions41 Regions

http://www.cs.berkeley.edu/projects/vision/grouping/segbench/


15


State-of-the-Art Segmentation (1)

Graph Cut [Shi+Malik, PAMI 2000]

J. Shi, J. Malik, “Normalized Cuts and Image Segmentation”, PAMI 22(8):888-905, 2000

Fully connected graph (all pixels of the image !)

Weights wpq measure similarity between pixel p, q


16


Graph cut (2)J. Shi, J. Malik, “Normalized Cuts and Image Segmentation”, PAMI 22(8):888-905, 2000

Partition the graph into subgraphs

Goal: High similarity within a subgraph

Low similarity between subgraphs


17


N-Cut Results (1)


18


N-Cut Results (2)


19



Graph Based [Felzenszwalb+Huttenlocher, IJCV 2004]P. Felzenszwalb, D. Huttenlocher, “Efficient, Graph-based Image Segmentation”, IJCV 59(2), 2004

Pairwise comparison of regions

Original image segmentation using segmentation using

“grid graph” “nearest neighbor graph”


20


Graph Based (2) [Felzenszwalb+Huttenlocher, IJCV 2004]

P. Felzenszwalb, D. Huttenlocher, “Efficient, Graph-based Image Segmentation”, IJCV 59(2), 2004

Pairwise comparison of regions

Original image segmentation using segmentation using

“grid graph” “nearest neighbor graph”


21



Level-Set Based SegmentationJames Sethian (1999): Level Set & Fast Marching Methods, Cambridge.

Stan Osher & Ronald Fedkiw (2002): Level Set Methods and Dynamic Implicit Surfaces, Springer.

Stan Osher & Nikos Paragios (2003): Geometric Level Set in Imaging, Vision+Graphics, Springer.

Split the image into 2 regions, such that:

• Similarity between pixels of a region is maximized

• Contour length is minimized

e.g.: [Chan&Vese, 2001]

dH

dHufHufuuE

)(

)(1)()()(,, 22


22


Level-Set Based Segmentation (2)

dH

dHufHufuuE

)(

)(1)()()(,, 22

u+,u- Average intensities fore- and background

Φ Level-Set function

Ω image domain

f image point (pixel)

H Heaviside step function

ν weight of regularisation term (contour)

contour

Minimize E w.r.t. , u+, u-, +, -

https://en.wikipedia.org/wiki/Heaviside_step_function


23



Φ>0

Φ<0

Φ>0

Φ<0

Φ>0

Φ<0

Many adaptations, various formulations

e.g. [Paragios&Deriche], [Brox&Weickert], [Fussenegger]


24



Multi-region Level-Set


25


Tokenset: Relaxing the formal definition

1. ∪ 𝑅𝑖 = 𝐼

2. 𝑖 ≠ 𝑗: 𝑅𝑖 ∩ 𝑅𝑗 = ∅

3. some homogeneity criterion H holds: ∀𝑖: 𝐻 𝑅𝑖 = 𝑡𝑟𝑢𝑒

4. disjunct neighbors: 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑅𝑖 , 𝑅𝑗 : 𝐻 𝑅𝑖 ∪ 𝑅𝑗 = 𝑓𝑎𝑙𝑠𝑒

• Tokens may be post-processed (e.g. opening/closing)• #of holes, #of parts several regions!

• Tokens may be overlapping (at least: their bounding boxes)

• No need to cover the whole image!

image

image

description

segmentation

2D grouping


26


Token – Image Object

• Points (x,y)

• Lines ((x1,y1),(x2,y2))

• Polylines/Chains ((x1,y1),(x2,y2), …,(xn,yn))

• Polygons ((x1,y1),(x2,y2), …,(xn,yn),(x1,y1))Squares, rectangles, circles, ellipses, … (parametrized closed contours)

• Constellations/Bitmaps

“Feret”-Box

Bounding box, aligned (x,y)

… foreground

… background


27


Constellation Tokens


28


Tokenset

A file – i.e., a list of tokens database, various indexingTo

ke

n T

yp

e

Lexicon

Data


29


Tokenset 2D Image/Scene Description

“houses” [Matsuyama’90] “face” [Brunelli’92] “pedestrians” [Suzuki’90]


30


Example: PhD Pinz (1988)

Finding trees in aerial images …

Original image Smoothed (conv. Gaussian) Local brightness maxima

Circles

circles trees

Spruce (Fichte) + Pine (Kiefer)


31


Example: PhD Pinz (1988)

Finding trees in aerial images …

Original image Smoothed (conv. Gaussian) Local brightness maxima

Circles

circles trees

Spruce (Fichte) + Pine (Kiefer)

2D image 2D scene

description

image

image

description

image proc.

segmentation

2D grouping

2D scene

description


32





“token”

“tokenset”


image

image

description

segmentation

2D grouping


33


Some very fundamental questions:

• What characterizes an object ?

“objectness”

- compact, (self-)similar, distinct (color, texture), …

• Given an object, what characterizes its shape ?

• Maybe easier to answer: What is not shape?

- color

- texture

- size

?


34


“Objectness”

[Alexe et al., PAMI 2012]

cf. “Region Proposal Network – RPN” as one component in ConvNets for Object

Detection (e.g., Faster R-CNN, CVPR’14, see https://arxiv.org/abs/1506.01497)

https://arxiv.org/abs/1506.01497


35


Object ShapeHumans (again!) are very good

in characterizing shape!

Proportions !

Pablo Picasso, rites of spring,

from D.Marr, VISION, fig. 3-56 (a)

2D 3D


36


Ideas open to further development

• Spatio-temporal shape (2D space + time)

• Motion patterns, trajectory space, …

• Spatio-temporal shape (3D space + time)


37


Token Features for 2D Grouping

Shape• Minimum bounding rectangle (MBR)

• Best ellipse fit

• Aspect ratio (AR): |log(height/width)| = |log(width/height)|

• BR fill: % of foreground pixels in Feret box or in MBR

• Circumference

• Compactness = 𝑎𝑟𝑒𝑎

𝑐𝑖𝑟𝑐𝑢𝑚𝑓𝑒𝑟𝑒𝑛𝑐𝑒2

• Elongatedness = 1 −𝑚𝑖𝑛𝑜𝑟 𝑎𝑥𝑖𝑠

𝑚𝑎𝑗𝑜𝑟 𝑎𝑥𝑖𝑠of the best ellipse fit

Appearance• Color

• Texture

• Histograms of … local descriptors


38


Homographies, Collineations, Perspective Transformations

vs.

Token shape, appearance, etc.

I borrow from „Bildgestützte Messverfahren“

(image-based measurement 2VO, 1LU) …

A hierarchy of transformations – hierarchy of geometries:

• Euclidean

• Similarity transformation

• Affine transformation

• Perspective transformation

3

2

1

333231

232221

131211

3

2

1

'

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

x

x

H


39


A Hierarchy of Transformations / Geometries

100

2221

1211

y

x

taa

taa

100

2221

1211

y

x

tsrsr

tsrsr

333231

232221

131211

hhh

hhh

hhh

100

2221

1211

y

x

trr

trr

Projective

8dof

P

Affine

6dof

A

Similarity

4dof

S

Euclidean

3dof

E

In 2D, a square transforms to:

3

2

1

333231

232221

131211

3

2

1

'

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

x

x

H

Invariance of token features?


40


Invariance of Token FeaturesRadiometric

transformation

E

Euclidean

S

Similarity

A

Affine

P

Projective

Feret box: AR

Feret: BR fill

MBR: AR

MBR fill

best ellipse

elongatedness

circumference

compactness

size (# pix.)

color

texture

# holes

# parts


41


Examples 2D GroupingLarge tokens:

Original image 434 constellation tokens 9 tokens > 2000 pixels

Parallelism

Original image 26002 straight lines 94 lines with orientation -4


42


Grouping – Example (KU/09)

edges circles (Hough)

coins


43


Perceptual Grouping (Lowe’87)

input image projection of

3D wireframe model

successful matches

by perceptual grouping


44


The Umass VISIONS SystemInterpretations of Massachusetts “road” scenes [Draper et al., IJCV 1989]

original image interpretation result interpretation key


45


The Umass VISIONS SystemInterpretations of Massachusetts “road” scenes [Draper et al., IJCV 1989]



46


The Umass VISIONS SystemInterpretations of Massachusetts “house” scenes [Draper et al., IJCV 1989]



47


Bottom-Up vs. Top-Down Grouping

2D Models

• Chains of “edgels”, “ridgels”

• Hough transform:

complex patterns local maxima

Image space “Hough” space, bins

• Active contour models

• Shape priors

• Active shape models


48


Bottom-Up vs. Top-Down Grouping

2D Models

Edges chains

Lines ridges

Lines valleys


49


“Snakes” – Active Contour Models

Kass, Witkin, Terzopoulos, 1st ICCV, London, 1987

Image energies (greylevel, gradient, …), mechanical model (spring)

Active contours adaptation to subjective contours


50


“Snakes” – Active Contour Models

Kass, Witkin, Terzopoulos, 1st ICCV, London, 1987

Tracking of moving contours

Extension to 3D:

“Balloons”


51


“Shape Priors” in Level Set Segmentation[Fussenegger]

… global deformation (scale, rot, translation)


52


“Active Shape Models” in Level Set Segmentation[Fussenegger]

• Shape changes dependent on viewpoint train ASM

• Videos level_set_hide+seek level_set_teapot


53


“Active Shape Models” in Level Set Segmentation[Fussenegger]

3 ASMs

learnt:

- elephant

- octopus

- african man

a original img

b segment.

c, d:

varying the

order of the

3 ASMs


54


Summary IVU_1 – IVU_4

• Vision• Neurophysiology

• Cognitive psychology

• Computational theory (Marr paradigm, representations, algorithms)

• Linear Filtering, Convolution

• Definition of terms, system model of image understanding• Visual recognition the “holy grail” of computer vision

• Segmentation and grouping 2D image/scene description

… recap %


55


Definition: Visual Recognition [Perona’09]

“The holy grail of Computer Vision”

Five tasks of “visual recognition”:

– Verification (is a “car” in the image?)

– Detection and localization (what is there? where?)

– Classification (n “beach” images, m “city” images)

– Naming (name and locate all objects in an image)

– Description: objects, actions, relations, etc.

(example “kissing” “scene understanding”)

Increasing complexity from top bottom

Image and Video Understanding: mostly 2D (+time) recognition

Image-based Measurement: 3D (+time) reconstruction

Co

mp

lexity


56


My Model of Image Understanding


57





“token”

“tokenset”


image

image

description

segmentation

2D grouping

What next?


58


Course Schedule 2016/17• Vision

• Neurophysiology

• Cognitive psychology

• Computational theory (Marr paradigm, representations, algorithms)

• Linear Filtering, Convolution

• Definition of terms, system model of image understanding

• Visual recognition the “holy grail” of computer vision

• Segmentation and grouping 2D image/scene description

• Object categorization

• Terms, goals, issues, …

• Signal processing: Fourier, Gabor

• Scale

• Object models

• CNNs for image and video understanding

Documents

2D Scene Representation and Description · Level-Set Based Segmentation James Sethian (1999): Level Set & Fast Marching Methods, Cambridge. Stan Osher & Ronald Fedkiw (2002): Level