View
265
Download
1
Category
Preview:
Citation preview
FTV (Free-viewpoint Television)
Masayuki TanimotoProfessor Emeritus at Nagoya University
Nagoya Industrial Science Research Institute Nagoya, Japan
February 22, 2013
APSIPA Distinguished Lecture 2013APSIPA Distinguished Lecture at Telecom SudParis
APSIPA Mission: To promote broad spectrum of research and education activities in signal and information processing in Asia Pacific
APSIPA Conferences: ASPIPA Annual Summit and Conference
APSIPA Publications: Transactions on Signal and Information Processing in partnership with Cambridge Journals since 2012; APSIPA Newsletters
APSIPA Social Network: To link members together and to disseminate valuable information more effectively
APSIPA Distinguished Lectures: An APSIPA educational initiative to reach out to the community
Introduction to APSIPA and APSIPA DL
1. Progress of Visual Media2. History of FTV Development3. Principle of FTV4. FTV Technologies and Demo5. Ray-Reproducing FTV6. International Standardization of FTV7. Contributions of FTV8. Conclusions
Outline
1. Progress of Visual Media
Super HDTV demonstrated at the World Expo 2005 in Nagoya(provided by NHK Science & Technical Research Laboratories)
“イ” was displayed on the world’s first electronic TV display developed by Dr. TakayanagiKenjiro in 1926.
Progress of Television in 80 Years
40 scanning lines 4000 scanning lines100 times
To transmit only a partial information (single view) of 3D space
Next Challenge for Visual Media
FTV (Free-viewpoint TV)
To transmit all information (all views) of 3D space
FTV (Free-viewpoint TV)
A visual media that allows users to view a 3D scene by freely changing the viewpoint as if they were there.
TVTV
Progress of Visual Media
More pixels
3DTV3DTV
More views
HDTVHDTV
FTVFTV
More views
UDTVUDTV
More pixels
Num
ber
of
view
s
HD-FTVHD-FTV UD-FTVUD-FTV
HD-3DTVHD-3DTV UD-3DTVUD-3DTV
Number of pixels
Number of Views in Visual Media
⋅⋅⋅⋅⋅⋅⋅
⋅
⋅⋅
FTV
2DTV
Stereoscopic TV
Auto-stereoscopic TV3DTV
1
2
several
infinite
2. History of FTV Development
Process
ing
Capture
Display
Free viewpoint camera
FTV (Free viewpoint TV)
Ray-space
1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 year
Free-viewpoint bird’s eye view system
The Matrix
Eye VisionLight field
MPEG-FTV/3DV
MPEG-MVC
MPEG-3DAV
100-camera system
128/48-camera systems
Coding / Interpolation FTV with Audio
FTV on single PC
FTV onPC Cluster
FTV on Mobile Player
[Nagoya University]
[Others]
[MPEG]
Development of FTV Technologies
Pioneering Works on Free Viewpoint
• Multi viewpoint– Movie “Matrix”(1999)
– CMU “Eye Vision” (2001)
• Free viewpoint– Nagoya Univ. “Bird’s Eye View System”(2000)
– Nagoya Univ. “FTV”(2002)
Displayed views are not camera views but generated views.We can set viewpoints anywhere.
FTV
Multi camera
Viewpoint
Captured views
Virtual viewpoint
Generated view
Real cameras
How to realize FTV
Rays
Some views are captured by cameras.Other views are generated by the integration and interpolation of rays.
Free Viewpoint
Makoto Sekitoh, Koichi Toyota, Toshiaki Fujii, Tadahiko Kimoto and Masayuki Tanimoto, “A Virtual Bird's-Eye View System Based on Real Image, ”Electronic Imaging and the Visual Arts 2000 Gifu, 8, pp.8-1‐8-7, October 2000.
Bird’s-Eye View System(The World’s First FTV, 2000)
160×120 pixels, 10 frames/sec
3. Principle of FTV
“ Multiview Imaging and 3TDTV,” IEEE Signal Processing Magazine, Vol.24, No.6, pp.10-21 (November 2007)
12
34
5
X
Y
Z
( , )x y
( , )θ ϕ
Visual Field(Horizontal)
Visual Field(Vertical)
Cameras
Position
Direction
Viewing Zone
2 3 4 5
1
u ( tan )= θ
xy
f x y u( , , )
Visual Field(Horizontal)
Visual Field(Vertical)
Viewing Zone
Ray-Space
45
32
1
Capture of FTV
Real Space
Ray-Space
Horizontalsection
x
u=tanθ
xθtan
y
x
u=tanθ
line detection+ interpolation
Ray-Space Interpolation
Ray-Space and Horizontal Cross-Section
xy
u
f x y u( , , )
Section 1
Section 2
Section Image 1
Section Image 2
Ray-Space
Generation of FTV Images
Camera Arrangements for FTV
(b) Circular/spherical arrangementfor convergent view
(a) Linear/planar arrangementfor parallel view
u
v
0°
180°
360°
The horizontal cross-sections of spherical ray-spacehave sinusoidal structures.
Ray-Space in Circular Camera Arrangement
0° 180° 360°
4. FTV Technologies and Demo
Multi-Camera
Captured Images
CorrectedImages
Free-Viewpoint
Images
ReconstructedImages
Ray-Space
Capture Correction Encoder Decoder2D/3DDisplay
Transmission ViewGenerationStorage
M views 1~N views
Configuration of FTV System
4.1 Capture
Capture and Processing Parts of FTV
100-Camera SystemLinear Planar
Circular
4.2 Correction
Before correction After correction
Geometry Correction
(Test sequence of MPEG-FTV : Pantomime)
Color Correction
Geometry: onColor: off
Geometry: onColor: on
4.3 Compression
View(camera)
Time(frame)
Conventional coding:temporal correlationMVC :temporal and interview correlation
MVC (Multi-view Video Coding)
4.4 View Generation
FTV Demo: AquariumOriginal Camera Views (15 Cameras)
FTV Demo: AquariumGenerated Free Views
Original Views
Generated Views - Interpolated -
Generated Views - Forward and Backward -
FTV on PC(Real-Time FTV on a laptop PC)
4.5 Display/User Interface
FTV on a laptop PC FTV on a mobile player FTV on an all‐aroundwith mouse control with touch panel control 3D display (Seelinder)
FTV on a 2D display with head tracking
FTV on a multi‐view 3D display with head tracking
FTV on a multi‐view 3D display without head tracking
Interface of FTV
FTV on a Mobile Player
2D Display with View Control
Multiview 3D Display
without eye tracking with eye trackingviewing zone: wide crosstalk: eliminated
viewing zone: narrow, periodical crosstalk: large
Super-Multiview 3D Display: The SeeLinder
4.6 FTV for Static Scenes
Captured Video
Generated Free-viewpoint Images
Actual Camera Movement
Virtual Camera Movement
Demo 1 (Indoor Scene)
Generated Free-viewpoint Images
Captured Video
Actual Camera Movement
Virtual Camera Movement
Demo 2 (Outdoor Scene)
5. Ray-Reproducing FTV
Ray-Reproducing FTV
It captures all rays in a real 3D space and reproduces the same rays at different place and time.Thus, it provides us the same viewing as in the real world.
Object
Double
hyperbolic mirror
Galvanometer mirrorHigh-speed
Camera Real image of object
Configuration of Mirror-Scan System
Mirror-Scan Ray Capture (55 degree)
High Frame-rate Camera
Metal Halide Lamp
Parabolic Mirror x 2
Object Lower
Focal Point
Optical Image
of Object
Rotating Mirror
High Frame-rate Camera
Rotating Mirror
(45 deg Slanted)
Motor
Upper
Focal Point
Object
Motor
Acrylic Board
All-Around Ray Capture by Mirror-Scan 1
Captured Views of All-Around Mirror-Scan System
Milk drop into water(Rotating mirror:30 rotations/s, High-speed camera:10,800 fps)
Pig
Rays of 2D Display
Projected objects
Only control of magnitude needed
Display
Real objects
2D display emits rays with equal magnitude in all directions
Objects
rays from display
rays from objectsDisplay pixels
Control of magnitude and direction needed
Ideal 3D display emits rays with different magnitude in each direction
Rays of Ideal 3D Display
move
Principle of Ray Control
1-dimensional light source array
move
Parallax barrier
Cylindrical Parallax Barrier(Fast Revolution:1800rpm)
1-D Light Source Array(LED array)(Slow Revolution:100rpm)
Structure of the SeeLinder
• Directions of the rays are scanned by revolution.
• Intensity of each light ismodulated in synchronization with revolution.
All-Around Ray-Reproducing Display: The SeeLinder
High Speed Camera
Display
Distortion Correction/Data Conversion
Metal Halide Lamp
Mirror-Scan
All-Around Ray-Reproducing FTV
Demo of All-Around FTV
6. International Standardization of FTV
We proposed FTV to MPEG in 2001 and have been promoting its standardization.As the first phase of FTV, MVC (Multi-view Video Coding) started in 2004 and was completed in 2009. MVC has been adopted by Blu-ray 3D.As the second phase of FTV, 3DV (3D Video) started in 2007 and currently in progress at Joint Collaborative Team of MPEG and ITU-T.
FTV Standardization in MPEG
2001/12
Proposal of FTV
2003/10
CfCon 3DAV
CfEon MVC
CfPon MVC
3DAV seminar
2002/12 time
2004/10
2005/07
MVC (moved to JVT in 2006/07)
Evaluation of proposals
2006/01
Start MVC (Multi-view Video Coding)
2007/07
Req. on FTV
3DV (moved to JCT-3V in 2012/07)
2004/03
3DAV
•Requirements on 3DAV•EEs on 3DAV
•CEs on MVC
2007/04Start MPEG-FTV
2001/12Start 3DAV
2009/05MVC completed
Targets converged on FTV
Second phase of FTV
First phase of FTV
2009/02Requirements on 3DV
History of FTV Standardization in MPEG
CfPon 3DV
2011/03
3D video
Omni-directional
FTV
Stereo TV
Stereo TV with depth
Topics Discussed at 3DAV
FTV
Converged on FTV andMVC startedin March 2004
6.1 MVC
First Phase of FTV in MPEG
MVCEncoder
Sender side Receiver side
Trans-mission
MVCDecoder
Standardized in 2009Adopted by Blu-ray 3D
MVC (Multi-view Video Coding)
multi-view
6.2 3DV
Second Phase of FTV in MPEG
Depth Estimation
3DVEncoder
Stereo camera
Sender side Receiver side Multi-viewdisplay
Trans-mission
3DVDecoder
View Generation
multi-view
multi-depth
3DV (3D Video)
MVD (multi-view + depth)
Capture Correction/Conversion Encoder Decoder Display
TransmissionSynthesis
Storage
Multi-Camera Captured Multi-View Images
Corrected Multi-View Images and Multi-View Depth
Free ViewpointImages
Reconstructed Multi-View Images and Multi-View Depth
Test sequences
Depth EstimationReference Software
FTV Reference Model
View SynthesisReference Software
①
②
③
Contributions of Nagoya Univ. to 3DV
FTV Test Sequenceshttp://www.tanimoto.nuee.nagoya-u.ac.jp/
kendo balloons
pantomime champagne_tower dog
Rena Akko & Kayo
Kendo and balloons are captured by moving camera array
Champagne_tower
Balloons
July 5, 200781st MPEG, Lausanne
July 18, 2012101st MPEG, Stockholm
Development of 3DV
Participants: 6Input documents: 3
Participants: 150Input documents: 170
FormatConversion(optional)
Decoder View Synthesis(VSRS or else)
or
Framework of CfP for 3DV Standardization
Encoder
3-view case
2-view case
PSNR Subjective Test
MVD (multi-view + depth)
Alternative 3D Format
MVC+D (encode view and depth independently)
3D-AVC (depth is used to encode view)
3D-HEVC (depth is used to encode view)
Standardization Tracks for Coding
MVD (Multi-View plus Depth)
GVD (Global View and Depth)
Warp
Data Formats for 3D
MV (multi-view)
Data Formats for Television
MVD (multi-view + depth)
GVD(Global View and Depth)
1 view+ 1 depth
1 view 2 views
[2DTV] [Stereoscopic TV] [3DTV]
MVD
Multi view
GVD
Global view
Multi depth
Global depth
left center right
Comparison between MVD and GVD
Comparison Between GVD-NISRI and GVD-NICT
Global view
Global depth
GVD‐NICT
left center right
Global view
Global depth
GVD‐NISRI
7. Contributions of FTV
Multiview has been popular in special sessions and special issues of international conferences and journals since 2007.
FTV has been listed in the fields of interest and session themes of international conferences and journals since 2009.
Creating New Research Fields
18 - 20 January, 2010San Jose Convention Center, San Jose, CA United States
Session 5: DIBR and FTV (Depth Image Based Rendering and Free Viewpoint Television) Session Chair: Janusz Konrad, Boston Univ.
Paper 7524-17Quality improving techniques for free-viewpoint DIBRPaper 7524-18Structured light-based high-accuracy depth imaging applied for DIBR in multiview3DTVPaper 7524-19Novel view synthesis with residual error feedback for FTV
IS&T/SPIE Electronic Imaging 2010 Stereoscopic Displays and Applications XXI
S1 Coding and Processing for Advanced 3DTV/FTV (Special oral)Chair: Masayuki Tanimoto, Nagoya University, Japan
S1-1 3D Video Coding Using Advanced Prediction, Depth Modeling,and Encoder Control MethodsHeiko Schwarz, Christian Bartnik, Sebastian Bosse, Heribert Brust,Tobias Hinz, Haricharan Lakshman, Detlev Marpe, PhilippMerkle,,Karsten Müller, Hunn Rhee, Gerhard Tech, Martin Winken, Thomas WiegandFraunhofer Institute for Telecommunications Heinrich Hertz Institute, Germany
S1-2 Coding of Multiple Video+Depth Using HEVC Technology and Reduced Representations of Side Views and Depth MapsMarek Domański, Tomasz Grajek, Damian Karwowski, Krzysztof Klimaszewski, Jacek Konieczny, Maciej Kurc, Adam Łuczak, Robert Ratajczak, Jakub Siast, Olgierd Stankiewicz, Jakub Stankowski, Krzysztof WegnerPoznań University of Technology, Poland
S1-3 A Framework of 3D Video Coding Using View Synthesis PredictionCheon Lee, Yo-Sung HoGwangju Institute of Science and Technology, South Korea
S1-4 Robust Joint Reconstruction in Compressed Multi-view ImagingQionghai Dai, Changjun Fu, Xiangyang Ji, Yongbing ZhangTsinghua University, China
S1-5 Three-Dimensional Television System Based on Integral PhotographyJun AraiJapan Broadcasting Corporation, Japan
S1-6 3D Visual System Using Ray-Based Image Sensors and Electronic Holography DisplayKenji Yamamoto, Yasuyuki Ichihashi, Takanori Senoh, Ryutaro Oi, Taiichiro KuritaNational Institute of Information and Communications Technology, Japan
Emerging Techniques for Next Generation Video/Image Coding‐ Content‐ and context‐adaptive coding techniques‐ Joint codec optimization w/ context information‐ Parallel‐friendly coding algorithms‐ Perceptual coding and quality assessment‐ Non‐traditional coding techniques‐ Representation and analysis of visual signalsAdvanced Techniques for Emerging 3D Videos‐ Representation, analysis and coding of 3D scenes‐Multi‐view video processing and coding‐ 3D videos and free viewpoint TV‐ 2D to 3D video conversion‐ Graphics representations and content creationVisual Communications‐ Video streaming and networking systems‐ Internet video and QoS issues‐ Video on demand and live video services‐Wireless and broadband videos‐ Secure and robust video transmissions
Systems and Techniques for Human Interaction‐Mobile visual search‐ Futuristic human‐machine interfaces‐Multimodal recognition for mixed reality‐ Natural image to 2D/3D virtual translation‐ Display design for mixed reality‐ Virtually real and mixed reality applications‐ Biomedical imagingEmbedded Systems and Architecture Implementations‐ Concurrent algorithm/architecture optimization‐ Embedded software for DSP, ASIP, VLIW, multi‐core CPU, GPU, etc.‐ Architecture based design on VLSI, FPGA, reconfigurable architectures, SoC, etc.Cloud Multimedia Systems, Applications and Services‐ Infrastructure for media storage/distribution & computing‐ Performance metrics for multimedia processing in clouds‐ Large‐scale multimedia content processing and analysis‐ Complex semantic multimedia computing in clouds‐Mobile applications and services over clouds
Contribution to FIFA World Cup
2022 FIFA World Cup Japan Bid CommitteePlan to deliver the excitement on soccer stadium to the world by FTV
free navigation
realistic viewing
Innovation of FIFA World Cup Viewing by FTV
Free-viewpoint Vision
Viewing by free navigation
Full Court 3D Vision
Realistic 3D viewing
http://www.dream-2022.jp/jp/our_bid/bid_book/
“Revolutionizing Football” Produced by2022 FIFA World Cup Japan Bid Committee
TV Asahi “Hodo Station” Introducing FTV on December 1, 2010
Report on the final situation on FIFA World Cup Voting in ZurichThe key of Japanese plan is application of FTV to the World Cup.
We proposed the concept of FTV and realized it by developing ray capture, processing and display technologies.FTV is the ultimate 3DTV and ranked as the top of visual media.The international standardization of FTV has been conducted in MPEG.FTV enables realistic viewing and free navigation of 3D scenes as planed for FIFA World Cup.
8. Conclusions
Recommended