Proxy Plane Fitting for Line Light Field Rendering

Presented by Luv KohliCOMP238

December 17, 2002

Group Teleimmersion

• Project in UNC’s Office of the Future group

• Goal: create a system for many-to-many video teleconferencing– Attempt to provide best approximate

view for all participants while maintaining sense of presence

Approach

• Use light field rendering techniques to create novel views of participants

• Participant’s viewpoint tends to be limited to eye level with small lateral motions– Motivation for using a 1D linear array

of cameras instead of a full 2D setup

Line Light Field

• Less data required, so real-time capture, transmission, and rendering possible

Multiple focal planes

• Current system only allows one focal plane– Background out of focus– Participants out of focus if they move

away from focal plane• Use plane fitting and multiple focal

planes instead

Plane fitting

• Determine a focal plane for each participant using stereo correlation of silhouette points

• Epipolar constraint:

(u,v)

C1 C2

Plane fitting (2)

• Segment foreground objects out• Find silhouette points• Use stereo correlation of silhouette

points to find approximate plane– Silhouette point in one image

corresponds to epipolar line in second image

Multiple focal planes

• Use techniques described in Dynamically Reparameterized Light Fields (Isaksen, McMillan, Gortler) paper

• Conceptually, shoot a ray through light field and determine which focal planes it intersects

• Choose focal plane closest to geometry in scene – use scoring function σ

Multiple focal planes: scoring

Focal plane on participants

Focal plane on background

Focal plane on front objects

Multiple focal planes

Multiple focal planes

Multiple focal planes

Issues and future work

• Implementation issues• Will plane-fitting and multiple focal

planes be viable in a real-time networking environment?

• Work can be done in parallel• Will this enhance the feeling of

immersion?

References

• "Creating Adaptive Views for Group Video Teleconferencing -- An Image-Based Approach " Ruigang Yang, Celso Kurashima, Andrew Nashel, Herman Towles, Anselmo Lastra, Henry Fuchs. Presented at International Workshop on Immersive Telepresence, December 6, 2002, Juan Les Pins, France.

• S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M.F. Cohen. The Lumigraph. In Proceedings of SIGGRAPH 1996, pages 43-54, New Orleans, August 1996.

References

• M. Levoy and P. Hanrahan. Light Field Rendering. In Proceedings of SIGGRAPH 1996, pages 31-42, New Orleans, August 1996.

• A. Isaksen, L. McMillan, and S. J. Gortler. Dynamically reparameterized light fields. Technical Report LCS-TR-778, Massachusetts Institute of Technology, May 1999.

References

• C. Kurashima, R. Yang, A. Lastra. Combining Approximate Geometry with View-Dependent Texture Mapping – A Hybrid Approach to 3D Video Teleconferencing. Presented at SIBGRAPI 2002, XV Brazilian Symposium on Computer Graphics and Image Processing (07 – 10 October 2002, Fortaleza, CE, Brazil).

• L. McMillan and Gary Bishop. Plenoptic Modeling: An Image-Based Rendering System. In Proceedings of SIGGRAPH 1995, pages 39-46, 1995.