Yizhou Yu

Texture-Mapping Real Scenes from Texture-Mapping Real Scenes from PhotographsPhotographs

Texture-Mapping Real Scenes from Texture-Mapping Real Scenes from PhotographsPhotographs

Yizhou YuComputer Science Division

University of California at Berkeley

Yizhou YuComputer Science Division

University of California at Berkeley

SIGGRAPH 2000 Course onImage-Based Surface Details

Yizhou Yu

Basic StepsBasic StepsBasic StepsBasic Steps

• Acquire Photographs

• Recover Geometry

• Align Photographs with Geometry

• Map Photographs onto Geometry

• Acquire Photographs

• Recover Geometry

• Align Photographs with Geometry

• Map Photographs onto Geometry

Yizhou Yu

Camera Pose EstimationCamera Pose EstimationCamera Pose EstimationCamera Pose Estimation

• Input– Known geometry recovered from photographs or laser

range scanners

– A set of photographs taken with a camera

• Output– For each photograph, 3 rotation and 3 translation

parameters of the camera with respect to the geometry

• Requirement– 4 correspondences between each photograph and the

geometry

• Input– Known geometry recovered from photographs or laser

range scanners

– A set of photographs taken with a camera

• Output– For each photograph, 3 rotation and 3 translation

parameters of the camera with respect to the geometry

• Requirement– 4 correspondences between each photograph and the

geometry

Yizhou Yu

Recover Camera Pose with Known Recover Camera Pose with Known CorrespondencesCorrespondencesRecover Camera Pose with Known Recover Camera Pose with Known CorrespondencesCorrespondences

• Least-squares solution– Needs good initial

estimation from human interaction

• Least-squares solution– Needs good initial

estimation from human interaction

i

yyi

xx iiiiqIqI 22 )()(

ip

iqiI

Camera

Image

) ( Tpq ii RP

ationDifferenti Numerical Gradient

Yizhou Yu

Recover Rotation Parameters only from Recover Rotation Parameters only from Known CorrespondencesKnown CorrespondencesRecover Rotation Parameters only from Recover Rotation Parameters only from Known CorrespondencesKnown Correspondences

• Constraints

• Least-squares solution

• Constraints

• Least-squares solution

ip

iq

Camera

Image

iv

in

T

0 ii nv R

i

ii nv 2) ( R

Yizhou Yu

Obtaining CorrespondencesObtaining CorrespondencesObtaining CorrespondencesObtaining Correspondences

• Feature Detection in 3D geometry and 2D images

• Human interaction– Interactively pick corresponding points in photographs

and 3D geometry

• Automatic Search– Combinatorial search

• Feature Detection in 3D geometry and 2D images

• Human interaction– Interactively pick corresponding points in photographs

and 3D geometry

• Automatic Search– Combinatorial search

Yizhou Yu

Automatic Search for Correspondences Automatic Search for Correspondences Automatic Search for Correspondences Automatic Search for Correspondences

• Pose estimation using calibration targets

• Combinatorial search for the best match– 4 correspondences each image

• Pose estimation using calibration targets

• Combinatorial search for the best match– 4 correspondences each image

3D Targets

Yizhou Yu

Camera Pose ResultsCamera Pose ResultsCamera Pose ResultsCamera Pose Results

• Accuracy: consistently within 2 pixels• Accuracy: consistently within 2 pixels

Texture-mapping a single image

Yizhou Yu

Texture MappingTexture MappingTexture MappingTexture Mapping

• Conventional texture-mapping with texture coordinates

• Projective texture-mapping

• Conventional texture-mapping with texture coordinates

• Projective texture-mapping

Yizhou Yu

Texture Map Synthesis ITexture Map Synthesis ITexture Map Synthesis ITexture Map Synthesis I

• Conventional Texture-Mapping with Texture Coordinates– Create a triangular texture patch

for each triangle

– The texture patch is a weighted average of the image patches from multiple photographs

– Pixels that are close to image boundaries or viewed from a grazing angle obtain smaller weights

• Conventional Texture-Mapping with Texture Coordinates– Create a triangular texture patch

for each triangle

– The texture patch is a weighted average of the image patches from multiple photographs

– Pixels that are close to image boundaries or viewed from a grazing angle obtain smaller weights

Photograph

Texture Map

3D Triangle

Yizhou Yu

Texture Map Synthesis IITexture Map Synthesis IITexture Map Synthesis IITexture Map Synthesis II• Allocate space for texture patches from

texture maps– Generalization of memory allocation to 2D

– Quantize edge length to a power of 2

– Sort texture patches into decreasing order and use First-Fit strategy to allocate space

• Allocate space for texture patches from texture maps– Generalization of memory allocation to 2D

– Quantize edge length to a power of 2

– Sort texture patches into decreasing order and use First-Fit strategy to allocate space

First-Fit

Yizhou Yu

A Texture Map Packed with Triangular A Texture Map Packed with Triangular Texture PatchesTexture PatchesA Texture Map Packed with Triangular A Texture Map Packed with Triangular Texture PatchesTexture Patches

Yizhou Yu

Texture-Mapping and Object ManipulationTexture-Mapping and Object ManipulationTexture-Mapping and Object ManipulationTexture-Mapping and Object Manipulation

Original Configuration

Novel Configuration

Yizhou Yu

Texture Map Compression ITexture Map Compression ITexture Map Compression ITexture Map Compression I

• The size of each texture patch is determined by the amount of color variations on its corresponding triangles in photographs.

• An edge detector (the derivative of the Gaussian) is used as a metric for variations.

• The size of each texture patch is determined by the amount of color variations on its corresponding triangles in photographs.

• An edge detector (the derivative of the Gaussian) is used as a metric for variations.

Yizhou Yu

Texture Map Compression IITexture Map Compression IITexture Map Compression IITexture Map Compression II

• Reuse texture patches– Map the same patch to

multiple 3D triangles with similar color variations

• K-means clustering to generate texture patch representatives

• Larger penalty along triange edges to reduce Mach Band effect

• Reuse texture patches– Map the same patch to

multiple 3D triangles with similar color variations

• K-means clustering to generate texture patch representatives

• Larger penalty along triange edges to reduce Mach Band effect

Texture Map

3D Triangles

Yizhou Yu

Synthetic Images with Compressed and Synthetic Images with Compressed and Uncompressed Texture MapsUncompressed Texture MapsSynthetic Images with Compressed and Synthetic Images with Compressed and Uncompressed Texture MapsUncompressed Texture Maps

Uncompressed20 texture maps

Compressed5 texture maps

20 texture maps 5 texture maps

Yizhou Yu

Projective Texture-MappingProjective Texture-MappingProjective Texture-MappingProjective Texture-Mapping

• Can directly use the original photographs in texture-mapping

• Visibility processing is more complicated

• Projective texture-mapping has been implemented in hardware, therefore, real-time rendering becomes possible

• View-dependent effects can be added by effectively using hardware accumulation buffer

• Can directly use the original photographs in texture-mapping

• Visibility processing is more complicated

• Projective texture-mapping has been implemented in hardware, therefore, real-time rendering becomes possible

• View-dependent effects can be added by effectively using hardware accumulation buffer

Yizhou Yu

Motivation for Visibility Processing: Motivation for Visibility Processing: Artifacts Caused by HardwareArtifacts Caused by HardwareMotivation for Visibility Processing: Motivation for Visibility Processing: Artifacts Caused by HardwareArtifacts Caused by Hardware

Camera

Image

GeometryTexture gets projected onto occluded andbackfacing polygons

Yizhou Yu

Visibility AlgorithmsVisibility AlgorithmsVisibility AlgorithmsVisibility Algorithms

• Image-space algorithms– Shadow buffer

– Ray casting

• Object-space algorithms– Weiler-Atherton

• Image-space algorithms– Shadow buffer

– Ray casting

• Object-space algorithms– Weiler-Atherton

Yizhou Yu

A Hybrid Visibility AlgorithmA Hybrid Visibility AlgorithmA Hybrid Visibility AlgorithmA Hybrid Visibility Algorithm

• Occlusion testing in image-space using Z-buffer hardware– Render polygons with their identifiers as colors

– Retrieve occluding polygons’ ids from color buffer

• Object-space shallow clipping to generate fewer polygons

• Occlusion testing in image-space using Z-buffer hardware– Render polygons with their identifiers as colors

– Retrieve occluding polygons’ ids from color buffer

• Object-space shallow clipping to generate fewer polygons

Yizhou Yu

Input Photographs and Recovered Input Photographs and Recovered Geometry from FacadeGeometry from FacadeInput Photographs and Recovered Input Photographs and Recovered Geometry from FacadeGeometry from Facade

Yizhou Yu

Visibility Processing ResultsVisibility Processing ResultsVisibility Processing ResultsVisibility Processing Results

The tower The rest of the campus

Yizhou Yu

Synthetic RenderingsSynthetic RenderingsSynthetic RenderingsSynthetic Renderings