Hylke Buisman and Derek ChanSupervisor: Christian Theobalt

Designing a 3D Video Camera

Real-time depth up-sampling

Hylke Buisman and Derek ChanSupervisor: Christian Theobalt

03-17-2008

Hylke Buisman and Derek ChanSupervisor: Christian Theobalt

Designing a 3D Video Camera

Problem description

• Primary goals– Super resolution and noise removal

• Input: low-res depth map, high-res color image• Output: 3D mesh of up-sampled and denoised depth

– Optimize for (near) real-time processing

• Data samples

Hylke Buisman and Derek ChanSupervisor: Christian Theobalt

Designing a 3D Video Camera

Approach

Bilateral filter affects only areas of similar color

Hylke Buisman and Derek ChanSupervisor: Christian Theobalt

Designing a 3D Video Camera

Approach

• Joint bilateral up-sampling (Kopf et al.):

• Implementations explored:– Bilateral grid (Paris & Durand 2007)– Separable kernel approximation

Hylke Buisman and Derek ChanSupervisor: Christian Theobalt

Designing a 3D Video Camera

Results

+ =

Hylke Buisman and Derek ChanSupervisor: Christian Theobalt

Designing a 3D Video Camera

Results

Naive Optimized Naïve (Color)

Bilateral grid (No color)

SeparableKernel (Color)

GPU

5 s 1 s 0.3 s 0.6 s 75 ms

Runtimes (800 x 600 image)

+

=

Input Result

Hylke Buisman and Derek ChanSupervisor: Christian Theobalt

Designing a 3D Video Camera

Hidden slide• Hylke - 50%

– Alignment:• Extrinsics calibration• Homography between depthmap and color image

– CPU implementation • JBU• Separable kernel

– Bilateral grid • Derek - 50%

– Alignment: • 3d point cloud tesselation• Reprojection into high-res camera

– Median filter– GPU implementation of upsampling– 3D results with texture