IMAGE VECTORIZATION USING GRADIENT MESHES
Jian Sun, Lin Liang, Fang Wen, Heung-Yeung Shum
Microsoft Research Asia
ACM SIGGRAPH 2007
Input image Initial mesh
Optimized gradient meshReconstruction
Cutout tool
Outline
Introduction Background Gradient Mesh Optimized Gradient Mesh Result Conclusions
Introduction
Image Vectorization
Goal : convert a raster image into a vector graphics Compact Scalable Easy to animate
Requirements Vector-based contents (eg. Flash or SVG) on
the Internet Vector-based GUIs used in Windows Vista
Gradient mesh
Gradient mesh, a drawing tool of commercial vector graphics editors
http://www.creativebush.com/tutorials/mesh_tutorial.php
Selecting mid value skin tone
Tracing photograph
Sampling colors from photo
Sampling colors from within the mesh
Start adding mesh points Sampling colors from face mesh to hide seam
Finished eye/eye socket
gradient mesh Image represented by gradient mesh
http://www.creativebush.com/tutorials/mesh_tutorial.php
Image vectorization tools
Adobe Illustrator, “Live Trace” Corel CoreDraw, “CorelTrace” AutoTrace, “AutoTrace”
Input image Adobe, Live Trace
Optimized gradient mesh Blend surface colors according to the
control points color as constructing surface by the control points
Optimize the gradient mesh as an energy minimization problem
Advantages Efficiency of use Easy to edit – modify, animation Scalability Compact representation
JPEG, 37.5 KB
Optimized, 7.7KB
Background
Object-based vectorization
Object-based vectorization [Price and Barrett 06] Hierarchically segmentation of object and
sub-objects by a recursive graph cut algorithm
Subdivide meshes until the reconstruct error is below a threshold
Input image Subdivision mesh
RaveGrid [Swaminarayan and Prasad 06]
Constrained Delaunay triangulation of the edge contour set
cont.
Ardeco Automatic Region Detection and
Conversion algorithm [Lecot and Levy 06] Cubic splines Each region filled with a constant color, or a linear or
circular gradient
cont.
Gradient Mesh
Overview
Input raster image
Initial mesh(Coons mesh)
Optimized gradient mesh
(Ferguson patch)
Reconstruction image
min arg Energy(Mesh) non-linear least squares (NULL) problem
Cutout tool
Levenberg-Marquardt (LM) algorithm
Color fitting: monotonic cubic spline interpolation
Coherent matting method: sample object boundary color from the estimated foreground colors
E(M) Constrains:1. Smoothness2. User guided vector3. Boundary
Mesh initialization (manually) Decompose the input image into sub-
objects using Cutout tool [Li et al. 04]
Divide each sub-objects with 4 segments manually as Coons patches
m(u,v): the position vector of a point (u,v) Q : control points el al. F : basis functions
Ferguson patch [Ferguson 1964]
P(0)=A P(1)=B
TA TB
B
A
T
T
B
A
uuuuP
1122
1233
0100
0001
1)( 32
Basic Curve Segmentation
Ferguson patch
Basic Surface Segmentation
0
Ferguson patch
Basic Surface Segmentation
0
Gradient mesh
A gradient mesh consists of topologically planar rectangular Ferguson patches with mesh-lines
For each point q Position: {xq,yq}
Derivatives: {mqu,mq
v, αqumq
u, αqvmq
v} RGB color: cq = {cq(r), cq (g), cq (b)}
TTf VCUCQvuf ),(
1122
1233
0100
0001
C
321 uuuU 321 vvvV
00
0031
20
3131
2020
uu
uu
vv
vv
f
CC
CC
CCCC
CCCC
Q
Ferguson patches are lack of Cv and Cu !
Color interpolation
[Wolberg and Alfy 99] Determine the smoothest possible curve that
passes through its control points and satisfy monotonic constraint
The seven data points are monotonically increasing in f(xi) for 0 i 6, the cubic spline is not monotonic≦ ≦
Monotonic cubic spline
Rendering Ferguson patches
1. Sample color of control points2. Estimate Cu, Cv by Monotoic Cubic
Spline algorithm3. Render Ferguson patches
TTf VCUCQvuf ),(
Scalability
A gradient mesh• original resolution (x
1)
Scaling result (x8)• shape edges are well
preserved
Bi-cubic raster scaling (x8)• Blocky artifacts
appear
Optimized gradient mesh
Minimize E(M)
min arg.
Solve NULL problem using LM algorithm
Minimizing E(M) is a non-linear least squares (NULL) problem Energy function
z: vector form of unknowns in M
Levenberg-Marquardt (LM) algorithm is the most successful solver for NULL [Levenberg 44], [Nocedal and Wright 99]
cont.
Gaussian pyramid from the input image and apply coarse-to-fine optimization for LM
Optimization
Optimized gradient mesh• 0.7/pixel reconstruction
error• 40 iterations
Gradient mesh of Adobe Illustrator
Smooth constraint
Input image Opt. gradient mesh without smooth(err. 1.8/pixel)
Opt. gradient mesh with smooth (err. 0.9 pixel)
vtus 5,5,50 Smooth neighboring patches also,mp(- s, t) = m△ p-1(1- s, t) △
Vector line guided optimized gradient mesh
Initial mesh
User guided vector, V
Opt. gradient mesh with user guided vector(err. 0.5/pixel)
Directly optimized gradient mesh (err. 2.5/pixel)
Vector line guided optimized gradient mesh
Initial mesh Opt. gradient mesh with V
w = 1/5 L
3wv
Boundary constraint
The boundary of a gradient – one or more cubic Bezier spline
The control points on the boundary only move along the splineEx: control point q on the spline S in u direction
Results
Red pepper
Gradient meshes by an artist (354
patches)
Initial gradient meshes
Optimized the highlight and
shadow regions are reconstructed
Sculpture
Input image ReconstructionOptimized
gradient mesh
Face
Input image ReconstructionOptimized
gradient mesh
Conclusions
Introduce the gradient mesh as an image representation tool first
Present optimized gradient mesh
Limitations A fine image details and
highly textured image Boundaries or topologies are
too complicated
Input image
Reconstructed image
Optimized gradient meshes
END