Upload
gabriel-taubin
View
1.149
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Brown University Spring 2012 CourseENGN2502 3D Photography
Citation preview
ENGN2502 3D Photography Spring 2012
Gabriel Taubin Brown University
What do we mean by 3D Photography ?
• Techniques and systems using cameras and lights to capture the shape and appearance of 3D objects
• The geometry of triangulation • Surface representations and data structures • Methods to smooth, denoise, edit, compress, transmit,
simplify, and optimize very large polygonal models • Applications to computer animation, game development,
electronic commerce, heritage preservation, reverse engineering, medicine, virtual reality, etc.
• Project oriented • Goal: publication quality final projects • Instructor permission required
3D Shape Capture
Representation / Data Structures
Simplification / Compression
Smoothing / Parameterization
Remeshing / Segmentation
Interactive Modeling
Optimization / Resampling
Surface Reconstruction
Out-of-Core Algorithms
ENGN2502 : 3D Photography *Spring’2012+
• Medical Imaging – Visualization – Segmentation
Motivation
• Industry – Reverse engineering – Fast metrology – Physical simulations
• Entertainment
– Animating digital clays for movies or games
• Archeology and Art – Digitization of cultural
heritage and artistic works
Laser range scanning devices Multi-camera systems
3D Shape and Appearance Capture
Structured lighting systems
http://mesh.brown.edu/byo3d/
Stereoscopic Photography
Screen shots of our real-time stereo system working on the field
Real-Time High-Definition Stereo on GPGPU using Progressive Multi-Resolution Adaptive Windows
Y. Zhao, and G. Taubin, Image and Vision Computing 2011.
¼ Resolution ½ Resolution Full Resolution¼ Resolution ½ Resolution Full Resolution
Coarse-to-fine matching on multiple resolutions
Stereo Frame Grabbing
Stereo Rectification
Multi-Resolution Pyramid Generating
Background Modeling With Shadow Removal Foreground Detection with Dilation & Erosion
Stereo Matching Using Adaptive Window with Cross-Checking
Disparity Refinement R
eso
luti
on
Sca
n f
rom
Lo
w t
o
Hig
h
Processing Pipeline
Real-Time High-Definition Stereo on GPGPU using Progressive Multi-Resolution Adaptive Windows
Y. Zhao, and G. Taubin, Image and Vision Computing 2011.
Time of Flight 3D Scanning
nssm
m
c
dt 17
103
0.528
Single Shot Structured Lighting: MS Kinect
3D Triangulation: Ray-Plane Intersection
ray
plane
intersection point
projector /
coordinate systems
Triangulation by Line-Plane Intersection
object being scanned
pqn
}0)(:{ pt qpnpP
projected light plane
p
illuminated point on object
}{ vqpL L
camera ray
Lq
v
intersection of light plane
with object
same
coordinate
system
Triangulation by Line-Line Intersection
object being scanned
}{ 2222 vqpL
camera ray
2q
2v
projected light ray
1q1v
}{ 1111 vqpL
p
lines may not intersect !
Triangulation and Scanning with Swept-Planes
Structured Lighting using Projector-Camera Systems
Gray Code Structured Lighting
3D Reconstruction using Structured Light [Inokuchi 1984] • Recover 3D depth for each pixel using ray-plane intersection • Determine correspondence between camera pixels and projector planes by
projecting a temporally-multiplexed binary image sequence • Each image is a bit-plane of the Gray code for each projector row/column
Point Grey Flea2 (15 Hz @ 1024 x 768)
Mitsubishi XD300U (50-85 Hz @ 1024 x 768)
Gray Code Structured Lighting
3D Reconstruction using Structured Light [Inokuchi 1984] • Recover 3D depth for each pixel using ray-plane intersection • Determine correspondence between camera pixels and projector planes by
projecting a temporally-multiplexed binary image sequence • Each image is a bit-plane of the Gray code for each projector row/column • Encoding algorithm: integer row/column index binary code Gray code
Point Grey Flea2 (15 Hz @ 1024 x 768)
Mitsubishi XD300U (50-85 Hz @ 1024 x 768)
Gray Code Structured Lighting Results
3D Photography in Android
Reconstruction Method
Surface Representation
Oriented Points
Positions & Normals Implicit Surface Polygon Mesh
Typical Surface Reconstruction Pipeline
SSD: Smooth Signed Distance Surface Reconstruction F. Calakli, G. Taubin, Computer Graphics Forum, 2011.
• A new mathematical formulation
• And a particular algorithm
• To reconstruct a watertight surface
From a static oriented point cloud
Particularly Good at Extrapolating Missing Data
3D Reconstruction & Analysis of Bat Flight Maneuvers
• 3D Reconstruction of Bat Flight Kinematics from Sparse Multiple Views, by A. Bergou, S. Swartz, and G. Taubin, and K. Breuer, 4DMOD, 2011.
• 3D Reconstruction and Analysis of Bat Flight Maneuvers from Sparse Multiple View Video, by A. Bergou, S. Swartz, S. K. Breuer, G. Taubin, BioVis, 2011.
• Falling with Style - The Role of Wing Inertia in Bat Flight Maneuvers, by A. Bergou, D. Riskin, G. Taubin, S. Swartz, and K. Breuer, Annual Meeting, Society for Integrative and Comparative Biology, 2011.
• Falling with Style-Bat Flight Maneuvers, by A. Bergou, D. Riskin, G. Taubin, S. Swartz, and K. Breuer, Bulletin of the American Physical Society, Vol. 55, 2010.
• Multiple synchronized 1000fps+ cameras
• Controlled environment (backdrop & illumination)
• Bats trained to land on landing pad
• Experiments with several species
How do we measure bats ?
• Bats have highly articulated wings • Very complex wing motion • Current goal: Detailed reconstruction of wing and body
kinematics and derivatives from visual data • Skeleton model with 52 degrees of freedom • Geometry parameterized by 37 constants • Future Goal: Model-less Dynamic Shape Reconstruction
Some Methods to Capture 3D Point Clouds
Shadow
Turntable
Backdrop
8 Megapixel Camera
Multi-Flash Attachment
Multi-Flash Camera
Beyond Silhouettes: Surface Reconstruction using Multi-Flash Photography
D. Crispell, D. Lanman, P. Sibley, Y. Zhao and G. Taubin [3DPVT 2006]
30
Shadow
Turntable
Backdrop
8 Megapixel Camera
Multi-Flash Attachment
Multi-Flash Camera
Multi-Flash 3D Photography:
Capturing the Shape and Appearance of 3D Objects
Turntable Rotation
A new approach for reconstructing 3D objects using shadows cast by depth discontinuities, as detected by a multi-flash camera. Unlike existing stereo vision algorithms, this method works even with plain surfaces, including unpainted ceramics and architecture.
Estimated Shape: 3D Point Cloud
Recovered Appearance: Phong BRDF Model
Multi-Flash Turntable Sequence: Input Image
Data Capture: A turntable and a digital camera are used to acquire data from 670 viewpoints. For each viewpoint, we capture a set of images using illumination from four different flashes. Future embodiments will include a small, inexpensive handheld multi-flash camera.
Recovering a Smooth Surface
The reconstructed point cloud can possess errors, including gaps and noise. To minimize these effects, we find an implicit surface which interpolates the 3D points. This method can be applied to any 3D point cloud, including those generated by laser scanners.
Raskar et al. [Siggraph 2004]
• Depth discontinuity estimation for Non-Photorealistic Rendering
– Camera static with respect to object
– 4 Images are captured with object
– Each illuminated by a different flash
– Flashes located close to camera lens
– Image processing extracts and combines shadows
What do we gain?
Using only silhouettes Using all depth discontinuities
Silhouettes vs Depth Discontinuities [2D]
Silhouette
Depth Discontinuity
Differential Shape From Silhouette [Cipolla & Blake 92]
Known from camera motion
Measured in epipolar slice
Measured in depth discontinuity image
Computing r’(t) : Depth Discontinuity Edge Tracking
1
2 3
4
1 2
3 4
VFIso Results [2006 110x110x110 grid]
40
Using the implicit surface, we can determine which points are visible
from each viewpoint. To model the material properties of the surface,
we fit a per-point Phong BRDF model to the set of visible reflectance
observations (using a total of 67 viewpoints).
Multi-Flash 3D Photography: Photometric Reconstruction
Multi-Flash Turntable Sequence
Images
Phong (Specular)
Phong (Diffuse) Estimated Phong Appearance Model
Diffuse Specular Ambient
…
3D Point
Cloud
Implicit Surface
Surround Structured Lighting: 3-D Scanning with Orthographic Illumination D. Lanman, D. Crispell, G. Taubin [CVIU 2009]
3D Slit Scanning with Planar Constraints M. Leotta, A. Vandergon, and G. Taubin [CGF 2008]
Camera 1
Camera 2 Laser pointer +
cylindrical lens
Catadioptric Stereo Implementation
Can Estimate Points Visible From One Camera
Surface Reconstruction from Multi-View Data
NSF Digital Archaeology Project
• The main goal is to automate the tedious processes of data collection and documentation at the excavation site, as well as to provide visualization tools to explore the data collected in the database
• Also to solve specific problems in Archaeology using computer vision techniques.
• We first used a network of cameras to capture the activity at the excavation site, to reconstruct the shape of the environment as it is being excavated, to reconstruct layers, and to locate finds in 3D
• Now we use multi-view stereo from photographs captured by a handheld digital camera
Semi-automatically Assemble Fragments Into Artifacts
Assisted Data Acquisition, Algorithmic Reconstruction, Integrated multi-format analysis
REVEAL Archaeological Data Acquisition
Objects: Artifacts
Excavations Areas Sites
Data Acquisition
Data: Text Photos Video 3D Models
Automatically Convert Photos to 3D Models
Site Plans
Import photos, videos, and laser scans and connect them to database objects
Improve speed and accuracy with computer assisted data entry
Advanced Algorithms
Import External Data
Laser Scanned Models
Rule-based Reconstructions
REVEAL Database
Typical Activity Sequence
Data integrated and synchronized in tabular, plan drawing, 3D spatial, image, and video formats
REVEAL Archaeological Analysis
Select Artifacts on Site Plan
Artifacts Excavations
Areas Sites
Display Photos of Selected Artifacts
Examine Relationship of Artifacts in-situ in auto-generated 3D Excavation Model
Export Formatted Artifact Data for inclusion in Site Publication
[Furukawa and Ponce 2008]
[Snavely et. al. 2006]
MVS software
Patch-based Multi-View Stereo (PMVS) http://grail.cs.washington.edu/software/pmvs/
http://phototour.cs.washington.edu/bundler/
Accurate 3D Footwear Impression Recovery From Photographs, F. A. Andalo, F. Calakli, G. Taubin, and S. Goldenstein,
International Conference on Imaging for Crime Detection and Prevention (ICDP-2011).
Comparable to 3D Laser Scanner
Challenges
Uniform sampling
Non-uniform sampling
Noisy data
Misaligned scans
From Multi-View Video Cameras
View Interpolation From Multi-View Video Cameras
View Interpolation From Multi-View Video Cameras
View Interpolation From Multi-View Video Cameras
View Interpolation From Multi-View Video Cameras
With Background Segmentation
Ongoing work
ENGN2502 3D Photography Spring 2012
I hope to see you in class !