2D Scene Representation and Description · Level-Set Based Segmentation James Sethian (1999): Level...

Institute of Electrical Measurement and Measurement Signal Processing

Axel Pinz WS 2017/18 Image and Video Understanding 4

2D Scene Representation and Description

You can get very far in 2D !

2D “image object”

“token”

“tokenset”

2D scene description

description

segmentation

2D grouping

Segmentation: From Images to Tokens

• “Formal” definition of segmentation:Image I is segmented segmentation S = {regions Ri | rules 1-4}

1. ∪ 𝑅𝑖 = 𝐼

2. 𝑖 ≠ 𝑗: 𝑅𝑖 ∩ 𝑅𝑗 = ∅

3. some homogeneity criterion H holds: ∀𝑖: 𝐻 𝑅𝑖 = 𝑡𝑟𝑢𝑒

4. disjunct neighbors: 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑅𝑖 , 𝑅𝑗 : 𝐻 𝑅𝑖 ∪ 𝑅𝑗 = 𝑓𝑎𝑙𝑠𝑒

• Fundamental concepts of segmentation• Region-based segmentation (e.g. threshold, split+merge, R growing)

• Contour-based segmentation (e.g. edge detection closing of gaps)

• State-of-the-art segmentation algorithms• Graph based segmentation

• Level set

description

segmentation

2D grouping

Segmenting 2D “Shapes”

Region-Based Segmentation (1)

Global threshold

e.g.: Region growing Seed cells

e.g.: Split and Merge

Edge-Based Segmentation (1)

General approach:

1) Smoothing

2) Edge detection

3) Filteringa) Eliminate short edges

b) Close small gaps

4) Obtain closed contours

Image Edge image

Real edge profile Real line profile

Edge: inflection point

Line: local extremum

Locate it:

Zero crossing of 1st / 2nd derivative !!

Noise smoothing needed convolution with a Gaussian!

D. Marr + E. Hildreth: LoG / DoG zero crossings [1978]

“Mexican hat” / “Sombrero” Operator

http://laurent-duval.blogspot.co.at/2014/09/cours-radial-basis-functions.html

∆ 𝐺 ∗ 𝐼 = (∆𝐺) ∗ 𝐼

LoG / DoG zero crossings

always produce closed contours!

Original images

T … threshold

Canny edge

Detector

Relaxing the formal definition: Superpixels

Compactness and/or size parameters control the segmentation

Homogeneity may be violated:

disjunct neighbors: 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑅𝑖 , 𝑅𝑗 : 𝐻 𝑅𝑖 ∪ 𝑅𝑗 = 𝑓𝑎𝑙𝑠𝑒… may be true!

Example: [Achanta et al., SLIC superpixels, PAMI 34(11):2274-2282, 2012]

SLIC Examples [Achanta et al. PAMI ]

Spatial segmentations

into “superpixels”

SLIC Examples [Achanta et al. PAMI ]

Spatio-temporal

segmentations

into “supervoxels”

Temporal Superpixels TSPs [Chang et al CVPR‘13]

High-Level Segmentation – The “Correct” one?

3 Regions24 Regions41 Regions

http://www.cs.berkeley.edu/projects/vision/grouping/segbench/

State-of-the-Art Segmentation (1)

Graph Cut [Shi+Malik, PAMI 2000]

J. Shi, J. Malik, “Normalized Cuts and Image Segmentation”, PAMI 22(8):888-905, 2000

Fully connected graph (all pixels of the image !)

Weights wpq measure similarity between pixel p, q

Graph cut (2)J. Shi, J. Malik, “Normalized Cuts and Image Segmentation”, PAMI 22(8):888-905, 2000

Partition the graph into subgraphs

Goal: High similarity within a subgraph

Low similarity between subgraphs

N-Cut Results (1)

N-Cut Results (2)

Graph Based [Felzenszwalb+Huttenlocher, IJCV 2004]P. Felzenszwalb, D. Huttenlocher, “Efficient, Graph-based Image Segmentation”, IJCV 59(2), 2004

Pairwise comparison of regions

Original image segmentation using segmentation using

“grid graph” “nearest neighbor graph”

Graph Based (2) [Felzenszwalb+Huttenlocher, IJCV 2004]

P. Felzenszwalb, D. Huttenlocher, “Efficient, Graph-based Image Segmentation”, IJCV 59(2), 2004

Pairwise comparison of regions

Original image segmentation using segmentation using

“grid graph” “nearest neighbor graph”

Level-Set Based SegmentationJames Sethian (1999): Level Set & Fast Marching Methods, Cambridge.

Stan Osher & Ronald Fedkiw (2002): Level Set Methods and Dynamic Implicit Surfaces, Springer.

Stan Osher & Nikos Paragios (2003): Geometric Level Set in Imaging, Vision+Graphics, Springer.

Split the image into 2 regions, such that:

• Similarity between pixels of a region is maximized

• Contour length is minimized

e.g.: [Chan&Vese, 2001]

dHufHufuuE

)(1)()()(,, 22

Level-Set Based Segmentation (2)

dHufHufuuE

)(1)()()(,, 22

u+,u- Average intensities fore- and background

Φ Level-Set function

Ω image domain

f image point (pixel)

H Heaviside step function

ν weight of regularisation term (contour)

contour

Minimize E w.r.t. , u+, u-, +, -

https://en.wikipedia.org/wiki/Heaviside_step_function

Many adaptations, various formulations

e.g. [Paragios&Deriche], [Brox&Weickert], [Fussenegger]

Multi-region Level-Set

Tokenset: Relaxing the formal definition

1. ∪ 𝑅𝑖 = 𝐼

2. 𝑖 ≠ 𝑗: 𝑅𝑖 ∩ 𝑅𝑗 = ∅

3. some homogeneity criterion H holds: ∀𝑖: 𝐻 𝑅𝑖 = 𝑡𝑟𝑢𝑒

4. disjunct neighbors: 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟 𝑅𝑖 , 𝑅𝑗 : 𝐻 𝑅𝑖 ∪ 𝑅𝑗 = 𝑓𝑎𝑙𝑠𝑒

• Tokens may be post-processed (e.g. opening/closing)• #of holes, #of parts several regions!

• Tokens may be overlapping (at least: their bounding boxes)

• No need to cover the whole image!

description

segmentation

2D grouping

Token – Image Object

• Points (x,y)

• Lines ((x1,y1),(x2,y2))

• Polylines/Chains ((x1,y1),(x2,y2), …,(xn,yn))

• Polygons ((x1,y1),(x2,y2), …,(xn,yn),(x1,y1))Squares, rectangles, circles, ellipses, … (parametrized closed contours)

• Constellations/Bitmaps

“Feret”-Box

Bounding box, aligned (x,y)

… foreground

… background

Constellation Tokens

Tokenset

A file – i.e., a list of tokens database, various indexingTo

Lexicon

Tokenset 2D Image/Scene Description

“houses” [Matsuyama’90] “face” [Brunelli’92] “pedestrians” [Suzuki’90]

Example: PhD Pinz (1988)

Finding trees in aerial images …

Original image Smoothed (conv. Gaussian) Local brightness maxima

Circles

circles trees

Spruce (Fichte) + Pine (Kiefer)

Example: PhD Pinz (1988)

Finding trees in aerial images …

Original image Smoothed (conv. Gaussian) Local brightness maxima

Circles

circles trees

Spruce (Fichte) + Pine (Kiefer)

2D image 2D scene

description

image proc.

segmentation

2D grouping

2D scene

description

“token”

“tokenset”

description

segmentation

2D grouping

Some very fundamental questions:

• What characterizes an object ?

“objectness”

- compact, (self-)similar, distinct (color, texture), …

• Given an object, what characterizes its shape ?

• Maybe easier to answer: What is not shape?

- color

- texture

- size

“Objectness”

[Alexe et al., PAMI 2012]

cf. “Region Proposal Network – RPN” as one component in ConvNets for Object

Detection (e.g., Faster R-CNN, CVPR’14, see https://arxiv.org/abs/1506.01497)

Object ShapeHumans (again!) are very good

in characterizing shape!

Proportions !

Pablo Picasso, rites of spring,

from D.Marr, VISION, fig. 3-56 (a)

Ideas open to further development

• Spatio-temporal shape (2D space + time)

• Motion patterns, trajectory space, …

• Spatio-temporal shape (3D space + time)

Token Features for 2D Grouping

Shape• Minimum bounding rectangle (MBR)

• Best ellipse fit

• Aspect ratio (AR): |log(height/width)| = |log(width/height)|

• BR fill: % of foreground pixels in Feret box or in MBR

• Circumference

• Compactness = 𝑎𝑟𝑒𝑎

𝑐𝑖𝑟𝑐𝑢𝑚𝑓𝑒𝑟𝑒𝑛𝑐𝑒2

• Elongatedness = 1 −𝑚𝑖𝑛𝑜𝑟 𝑎𝑥𝑖𝑠

𝑚𝑎𝑗𝑜𝑟 𝑎𝑥𝑖𝑠of the best ellipse fit

Appearance• Color

• Texture

• Histograms of … local descriptors

Homographies, Collineations, Perspective Transformations

Token shape, appearance, etc.

I borrow from „Bildgestützte Messverfahren“

(image-based measurement 2VO, 1LU) …

A hierarchy of transformations – hierarchy of geometries:

• Euclidean

• Similarity transformation

• Affine transformation

• Perspective transformation

333231

232221

131211

A Hierarchy of Transformations / Geometries

333231

232221

131211

Projective

Affine

Similarity

Euclidean

In 2D, a square transforms to:

333231

232221

131211

Invariance of token features?

Invariance of Token FeaturesRadiometric

transformation

Euclidean

Similarity

Affine

Projective

Feret box: AR

Feret: BR fill

MBR: AR

MBR fill

best ellipse

elongatedness

circumference

compactness

size (# pix.)

texture

# holes

# parts

Examples 2D GroupingLarge tokens:

Original image 434 constellation tokens 9 tokens > 2000 pixels

Parallelism

Original image 26002 straight lines 94 lines with orientation -4

Grouping – Example (KU/09)

edges circles (Hough)

Perceptual Grouping (Lowe’87)

input image projection of

3D wireframe model

successful matches

by perceptual grouping

The Umass VISIONS SystemInterpretations of Massachusetts “road” scenes [Draper et al., IJCV 1989]

original image interpretation result interpretation key

The Umass VISIONS SystemInterpretations of Massachusetts “road” scenes [Draper et al., IJCV 1989]

The Umass VISIONS SystemInterpretations of Massachusetts “house” scenes [Draper et al., IJCV 1989]

Bottom-Up vs. Top-Down Grouping

2D Models

• Chains of “edgels”, “ridgels”

• Hough transform:

complex patterns local maxima

Image space “Hough” space, bins

• Active contour models

• Shape priors

• Active shape models

Bottom-Up vs. Top-Down Grouping

2D Models

Edges chains

Lines ridges

Lines valleys

“Snakes” – Active Contour Models

Kass, Witkin, Terzopoulos, 1st ICCV, London, 1987

Image energies (greylevel, gradient, …), mechanical model (spring)

Active contours adaptation to subjective contours

“Snakes” – Active Contour Models

Kass, Witkin, Terzopoulos, 1st ICCV, London, 1987

Tracking of moving contours

Extension to 3D:

“Balloons”

“Shape Priors” in Level Set Segmentation[Fussenegger]

… global deformation (scale, rot, translation)

“Active Shape Models” in Level Set Segmentation[Fussenegger]

• Shape changes dependent on viewpoint train ASM

• Videos level_set_hide+seek level_set_teapot

“Active Shape Models” in Level Set Segmentation[Fussenegger]

3 ASMs

learnt:

- elephant

- octopus

- african man

a original img

b segment.

varying the

order of the

3 ASMs

Summary IVU_1 – IVU_4

• Vision• Neurophysiology

• Cognitive psychology

• Computational theory (Marr paradigm, representations, algorithms)

• Linear Filtering, Convolution

• Definition of terms, system model of image understanding• Visual recognition the “holy grail” of computer vision

• Segmentation and grouping 2D image/scene description

… recap %

Definition: Visual Recognition [Perona’09]

“The holy grail of Computer Vision”

Five tasks of “visual recognition”:

– Verification (is a “car” in the image?)

– Detection and localization (what is there? where?)

– Classification (n “beach” images, m “city” images)

– Naming (name and locate all objects in an image)

– Description: objects, actions, relations, etc.

(example “kissing” “scene understanding”)

Increasing complexity from top bottom

Image and Video Understanding: mostly 2D (+time) recognition

Image-based Measurement: 3D (+time) reconstruction

lexity

My Model of Image Understanding

“token”

“tokenset”

description

segmentation

2D grouping

What next?

Course Schedule 2016/17• Vision

• Neurophysiology

• Cognitive psychology

• Computational theory (Marr paradigm, representations, algorithms)

• Linear Filtering, Convolution

• Definition of terms, system model of image understanding

• Visual recognition the “holy grail” of computer vision

• Segmentation and grouping 2D image/scene description

• Object categorization

• Terms, goals, issues, …

• Signal processing: Fourier, Gabor

• Scale

• Object models

• CNNs for image and video understanding

2D Scene Representation and Description · Level-Set Based Segmentation James Sethian (1999): Level...

Documents

A Variational Approach to Eulerian Geometry Processinggeometry.caltech.edu/pubs/MMTD07.pdf · vision, image processing, as well as graphics [Sethian 1999; Osher and Fedkiw 2003] since

Partial Differential Equations in Image and Surface Processingjhorak/evolene/Rumpf_pres.pdf · Osher, Fedkiw, Level Set Methods and dynamic implicit surfaces Springer, 2003 Sethian,

Level%SetMethods(&(Dynamic( ImplicitSurfaces(faculty.missouri.edu/.../lecture-notes/3-level-set.pdf · • S. Osher(and(R.(Fedkiw,(Level(SetMethods(and(Dynamic(• J.A.(Sethian,(Level(SetMethods(and(FastMarching(•

SETHIAN GNOSTICISM

A Fast Level Set Method with Particle Correction on ... · The level set method, devised by Osher and Sethian in 1988, is a powerful approach for tracking moving interfaces and widely

Narrow Band Level set on GPU - Kentzwang/schedule/zq9_1.pdfImplicit Contour: Level-set [1]S. Osher and R.P. Fedkiw, Level set methods and dynamic implicit surfaces, Springer, 2003

Nonlinear simulation of tumor growth via a level set method › ~sethian › 2006 › Papers › ... · more difficult the computational simulations, since a continuum model will

Practical Animation of Liquids - Computer graphicsgraphics.stanford.edu/~fedkiw/papers/stanford2001-02.pdf · 1 Practical Animation of Liquids Nick Foster* PDI/DreamWorks Ronald Fedkiw**

A Coupled Level Set Projection Method Applied to …sethian/2006/Papers/sethian...A Coupled Level Set Projection Method Applied to Ink Jet Simulation Jiun-Der Yu∗ Epson Research

Sethian Gnosticism by Joan Lansberry

Theory, Algorithms, and Applications Level Set Methods for ...math.berkeley.edu/~sethian/2006/Papers/sethian.actanumerica.1995.pdf · Level Set Methods for Propagating Interfaces

A semi-Lagrangian particle level set finite element method ... · J. A. Sethian, Level Set Methods and Front Marching Methods. Cambridge University Press (1999). S. Osher and R. Fedkiw,

A coupled level set–boundary integral method for moving boundary simulationsmath.berkeley.edu/~sethian/2006/Papers/sethian.breaking... · 2006. 8. 10. · water wave motion are

Crystal Growth and Dendritic Solidiﬁcation James A. Sethian …sethian/2006/Papers/sethian.crystal.pdf · Crystal Growth and Dendritic Solidiﬁcation James A. Sethian Department

Efficient Topology-Controlled Sampling of Implicit Shapespeople.csail.mit.edu/jchang7/pubs/posters/2012-ICIP.pdf · [10] Osher, S., Fedkiw, R.: Level Set Methods and Dynamic Implicit

Nonlinear simulation of tumor growth via a level set methodmath.berkeley.edu/~sethian/2006/Papers/sethian.angiogenesis.fdmp.2005.pdfComputational modeling of solid tumor evolution

Level Sets Methods in Imaging Science · • ”Level set methods add dynamics to implicit surfaces.” (Osher & Fedkiw book) • useful in analyzing and computing the motion of an

Level set methods for simulating multiphase displacements ...s-skj/Mod2FasePorNettVerk.06/Papers... · Level Set Methods ›Reference textbooks: •Osher, S. and Fedkiw, R.: «Level

Surface Propagation Methods - EMPossible · Level Set Method Fedkiw, Stanley OsherRonald, and Stanley Osher. "Level set methods and dynamic implicit surfaces." Surfaces 44 (2002):

A Hybrid Particle Level Set Method for Improved Interface ...graphics.stanford.edu/~fedkiw/papers/stanford2001-04.pdf · A Hybrid Particle Level Set Method for Improved Interface