CS294-69 Image Manipulation and Computational...

Tim Althoff, Philip Ly 1

Region-Filling and Object Removal by Exemplar-Based Inpainting. Criminisi, Pérez, and Toyama. CVPR 2004 Image Completion with Structure Propagation. Sun, Yuan, Jia and Shum SIGGRAPH 2005

CS294-69 Image Manipulation and Computational Photography

Tim Althoff, Philip Ly 2

Problem

Region filling, object removal, image completion, …

Tim Althoff, Philip Ly 3

Another Example

Tim Althoff, Philip Ly 4

Key ideas (1)

Exemplar-based synthesis

+ execution speed

+ accuracy of propagated structures

Tim Althoff, Philip Ly 5

Key ideas (2)

Filling order is critical

Tim Althoff, Philip Ly 6

Algorithm

1) Computing patch priorities at fill front → see next slide

2) Find patch with max priority 3) Find most similar exemplar (source region)

→ SSD in CIE lab color space 4) Copy image data & update confidences

Tim Althoff, Philip Ly 7

Patch priorities

P(p) = C(p) * D(p) Confidence term: amount of reliable information

surrounding pixel p Data term: encourages linear structures to be

synthesized first

Tim Althoff, Philip Ly 8

Limitations

Need similar patches for synthesis Only linear structures (curves?)

Cannot handle

depth ambiguities

Tim Althoff, Philip Ly 9

Competitive work & Impact

Diffusion-based techniques [Bertalmio et al., 2000]

No blur / faster → Patch-based approach wide-spread

Tim Althoff, Philip Ly 10

Image completion with Structure Propagation

Previous approaches have difficulties with salient structures in unknown regions

→ Structure propagation along user-specified curves

Tim Althoff, Philip Ly 11

Key Ideas

Synthesis ordering − Salient structures first − Then texture propagation into other regions

→ Graph labeling problem

Tim Althoff, Philip Ly 12

Key Ideas (2)

Formulate structure propagation as global optimization problem

− Similar structure − Overlapping patches

should match

Solve effectively using

− Dynamic Programming − Belief Propagation

Tim Althoff, Philip Ly 13

Results & Competitive Work

Criminsi et al., 2003 Sun et al., 2005

Tim Althoff, Philip Ly 14

Discussion

Questions?

Tim Althoff, Philip Ly 15

Discussion

Differences? − Combination structure and textures vs.

separation of structure and texture − Linear vs. curves − Amount of user input

Tim Althoff, Philip Ly 16

Discussion

Limitations? − Depth − Areas for sampling − Guidance by user

Tim Althoff, Philip Ly 17

Discussion

Solutions? − Layers and Bayesian Matting − Patch Matching

Tim Althoff, Philip Ly 18

Discussion

Future work? − Applying Belief Propagation to other graphical

work − Video/Meshes