Super-Resolution Texturing for Super-Resolution Texturing for Online Virtual GlobesOnline Virtual Globes

Diego Rother, Lance Williams and Guillermo SapiroDiego Rother, Lance Williams and Guillermo SapiroUniversity of Minnesota and Google, Inc.University of Minnesota and Google, Inc.

Internet Vision Workshop (CVPR 2008)

Online Virtual Globes

Online Virtual Globes

ClipMap pyramid.

Server Client

Requests

Tiles

• Problems:– Huge storage space required (one of the largest organized collections of imagery on the Internet).– Expensive acquisition and high transmission bandwidth.– Interpolation beyond available resolution: unnatural.

User

Earth surface

– Stereotypical. – Rapidly changing.– Identity important, details not.

Proposed Solution

• Proposition:– Synthesize, on the client, details for the

lower pyramid levels.– Using super-resolution techniques.– Harnessing labels and textures samples

from users (wiki model).

• Requirements:– Fast.– Seamless transition between layers.

Results

User input 1: Labels

Using interactive segmentation as in:Bai, X. and Sapiro, G., "A geodesic framework for fast interactive image and video segmentation and matting." ICCV, 2007.

Original frame User provided labels

Class1

(grass)

Class2

(path)

User input 2: Keyframed Texture

Keyframe1

Keyframe2

Keyscale1

Keyscale2

• User provides the texture pyramid:

Texture pyramid.

e.g., in meters/pixel

Synthesis of a New Layer

Input:from Server

Output:New Layer

Input:from Users

ClipMap pyramidLabels

Texture pyramid.

System Overview

Undo Color Matching

Texture Transfer

Color Matching

Interpolation

Interpolation

Pyramid of Training Textures

(from users)

Inputs

Last

Lay

er(fr

om s

erve

r)La

bels

(from

use

rs)

Outputs

New

Lay

erN

ew L

abel

s

Selection of the Training Image

Texture transfer: 1st passTraining Texture (from the texture pyramid)

Color matched image, without high frequencies

• Wei, L. and Levoy, M., "Fast Texture Synthesis using Tree-structured Vector Quantization." SIGGRAPH, 2000.• Efros, A. A. and Leung, T. K., "Texture Synthesis by Non-parametric Sampling." ICCV, 1999.

Small contexts → Fast

Y Channel(luminance)

I and Q Channels (chrominance)

Mean and Gradient Only Mean

Similarity between contexts considers

Source Locations

Produces 1-Pixel Patches:Contiguous areas copied verbatim from the training texture.

Texture transfer: 2nd, 3rd and 4th passes

• Ashikhmin, M., "Synthesizing Natural Textures." ACM Symposium on Interactive 3D Graphics. 2001.

Few candidates → Fast

Training TextureColor

ChannelPass

2nd

3rd and 4th

Y Channel(luminance)

I and Q Channels (chrominance)

Similarity between contexts considers

Produces Bigger Patches

Texture transfer from the same texture

Texture Pyramid

1st synthetic frame

2nd synthetic frame

• Patch interior (lilac and violet) directly copied.

• Patch boundaries (pink) synthesized in 4 passes as before.

• Doubles the patch size.

ClipMap Pyramid

Results: Maracanã

No texture transferred

No texture transferred

No texture transferred

No texture transferred

Texture pyramid (user input)

ClipMap pyramid (result)

Results: Maracana

No texture transferred

No texture transferred

No texture transferred

No texture transferred

ClipMap pyramid (result)

Source Locations (result)

Results: Field

No texture transferred

No texture transferred

No texture transferred

No texture transferred

No texture transferred

No texture transferred

No texture transferred

No texture transferred

No texture transferred

No texture transferred

ClipMap pyramid (result)

Texture pyramid (user input)

Source Locations (result)

Results: BeachClipMap pyramid (result)

Conclusions

• Proposed solution:– Reduces storage, bandwidth, and acquisition costs.– Improves appearance and information content.– Is fast (low dimensional contexts).