Stereoscopic Media Editing based on 3D Cinematography Principles

Chun-Wei Liu1 Tz-Huan Huang1 Ming-Hsu Chang1 Ken-Yi Lee1 Chia-Kai Liang2 Yung-Yu Chuang1

1National Taiwan University Refocus Imaging Inc.2

1 Introduction

Developments of stereoscopic displays and binocular cam-eras have made capture and display of stereoscopic mediaeasy. There will be strong needs for stereoscopic media pro-cessing to crop, make transitions, resize or stabilize stereo-scopic imagery as for conventional 2D media. However, de-spite of fast progresses on hardware, few have been made to-wards the stereoscopic media processing side, especially forstereoscopic media captured by consumers. This paper in-troduces 3D cinematography principles and applies them tostereoscopic media processing to maintain viewing comfortand pleasure. Stereoscopic video stabilization and stereo-scopic photo slideshows are used as examples to demonstratethese principles’ usage in stereoscopic media processing.

2 Principles and applications

3D cinematography principles. 3D filmmakers havelearnt and introduced several 3D cinematography principlesto reduce viewing discomfort and to enhance the immersionexperience when watching 3D films. Some of these principlesare not only useful for professional filmmaking but also forstereoscopic media processing [Mendiburu 2009]. (1) Main-taining coordination among views. The left and right imagesshould be processed in perfect coordination to avoid desyn-chronization and mismatch between views. (2) Having acontinuous depth chart. If the depth brackets of two neigh-boring shots are too far apart, viewers are forced to readjusttheir convergence from one shot to another. In this case, thestereopsis will be interrupted and the suspension of disbeliefis disturbed. (3) Placing rest areas between strong 3D shots.Viewers could experience eye strain if they stare at strong3D effects for too long. Thus, strong 3D shots should be in-terspersed with low 3D sequences. (4) Using shallow depthof field for shots with excessive depth brackets. If the fore-ground and the background are too far away from each other,the excessive depth range could make depth fusion difficult.Shallow depth of field can be used to isolate characters andto draw audience’s attention on the main character.Stereoscopic video stabilization. Independent applica-tion of conventional 2D video stabilization on both videostreams completely ignores the coordination and synchro-nization between two views. The results could bear theunwanted vertical parallax and inconsistent horizontal dis-parities. The former destroys the depth perception as theleft and right views are not perfectly horizontally alignedany more. The later leads to shimmering artifacts. The in-consistent time-varying horizontal disparities are interpretedas time-varying depths by our brains. Thus, viewers per-ceive that objects move forward and backward arbitrarily.Therefore, it is important to obey the constraints of hori-zontal alignment and consistent disparity during stabiliza-tion. Our method achieves stereoscopic video stabilizationby incorporating these constraints into the optimization pro-cess of Lee et al.’s 2D video stabilization method [Lee et al.2009]. Our method first extracts robust feature trajectoriesfrom the input video and builds correspondences of features

Figure 1: After stabilization, the video maintains horizon-tal alignment and consistent disparity (left), and becomesstabilized as the trajectories are smooth in time (right).

012345

Stabilization Depth continuity

Stereo Experience Acceptance

Aver

age

scor

e

Deshaker ICCV09 Proposed

0

1

2

3

4

5

Smoothness Depth continuity Stereo Experience Acceptance

Aver

age s

core

No Effect Fade-in/Fade-outBlending Depth ShiftDepth Shift +Fade-in/Fade-out Depth Shift + Blending

Figure 2: Results of user studies for stereoscopic video sta-bilization (left) and stereoscopic photo slideshows (right).

across the left and right views. Optimization is then per-formed to find a set of transformations to smooth out thesetrajectories while obeying these constraints.Stereoscopic photo slideshows. The proposed stereo-scopic slideshow system works as follows. (1) Optionally,shuffle the photo sequence so that the depth chart is morecontinuous if it is allowed to change the display order ofimages. For this, we have developed a photo shuffling al-gorithm to reorder photos so that neighboring photos havesimilar depth ranges. (2) Within display of an image, if thedepth bracket of the image is excessive, apply shallow depthof field blur filtering to gradually direct viewer’s attentionfrom the foreground to the background. For bringing only apart of the image to focus, each pixel is blurred according toits estimated depth to achieve the depth of field effects. (3)During transitions between two images, if there exists a largedepth jump between the in image and the out image, applythe active depth cuts trick. In addition, when switching fromthe out image to the in image, use the fade-in/fade-out ef-fects which gradually switch from the out image, to a blankimage and then to the in image.

Figure 1 gives an example for stereoscopic video stabi-lization. User studies show that, by incorporating 3D cine-matography principles, our methods provide more comfort-able and enjoyable 3D viewing experiences (Figure 2).

References

Lee, K.-Y., Chuang, Y.-Y., Chen, B.-Y., and Ouhyoung, M.2009. Video stabilization using robust feature trajectories. InProceedigns of the IEEE ICCV, 1397–1404.

Mendiburu, B. 2009. 3D movie making: stereoscopic digitalcinema from script to screen. Focal Press.