ETSI STQ: Workshop May 2017 Multichannel VR Audio ... · PDF fileMultichannel VR Audio Rendering to Stereo; Technique and Quality concerns ... (action camera, smartphone, ... •Great

ETSI STQ: Workshop May 2017 –

Multichannel VR Audio Rendering to

Stereo; Technique and Quality concerns

Fredrik Stenmark

Speech and Multimedia R&D, Qualcomm UK Ltd.

April 26, 2017

Prepared with contributions from Nils Peters, Akramus Salehin and Shankar Thagadur Shivappa

@2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. 2

Agenda

Auditory Scene

Capture

Channel / Object / Scene

Based Audio

Ambisonics

Audio

Scene

Rendering

Quality of

Experience

1 2 3 4 5


Auditory Scene CaptureStereo Capture

ORTF – Office de Radiodiffusion-Television Francais, HATS – Head And Torso Simulator

ORTF Stereophony Binaural HATS


Auditory Scene Capture

• Pros:

− Good Left/Right separation, precise localization left to right

− If HATS with HRTF used, can also localize above/below and front/back

− Relates closely to human hearing, places listener as a viewer in one location

• Cons:

− Lack of immersion, viewer of the scene as opposed to immersed in the scene

− Sounds appear from a front/back plane, turning the head has no effect

− Stereo capture optimized for stereo playback; other speaker configurations degrades the quality

Stereo Capture


Auditory Scene CaptureMulti-channel Capture

4-ch. Tetra-mic (Core Sound) n-ch. Eigenmike® (MH Acoustics)


Auditory Scene Capture

• Pros:

− Good sense of immersion, sounds appear from distinct

directions

− Localizes all X-Y-Z sources well generally

− 32 microphones in a small space captures the sound field

with high spatial resolution; the distance between mics

determine the low frequency accuracy and the high

frequency aliasing point

• Cons:

− More channel memory to store

− More expensive equipment

Multi-channel Capture


Overview Channel/Object/Scene Based Audio

Object-based Audio

Content Creator

Loudspeaker signals, e.g., 7.1+4MixingSound Object

Signal & Position

Main Microphone Exact speaker

configuration

necessary

Object

RenderingSound Object

Signal & Position

Content CreatorComplexity of rendering

increases with number of

objects in the scene

Sound Object

Signal & Position

HOA

Mixing

Content Creator

Ambisonics MicrophoneHOA

Rendering

Channel-based Audio

Scene-based Audio

HOA Renders to any number of

sound sources; but the reproduced

spatial resolution is only as good as

the captured resolution

(c)

20

06

, Ble

nd

er

Fo

un

da

tio

n /

Ne

the

rla

nd

s M

ed

ia A

rt In

sti

tute

/ w

ww

.ele

ph

an

tsd

rea

m.o

rg

Content Creator

Loudspeaker signals, e.g., 7.1+4MixingSound Object

Signal & Position

Main Microphone Home theatre system

With exact speaker

configuration

Channel Based Rendering

http://www.elephantsdream.org/

Object

RenderingSound Object

Signal & Position

Content CreatorComplexity of rendering

increases with number of

objects in the scene

Object Based Rendering

Sound Object

Signal & Position

HOA

Mixing

Content Creator

Ambisonics MicrophoneHOA

Rendering

Scene Based Rendering


Ambisonics Audio

− A basic Ambisonics Decoder is similar to virtual microphones,

and theoretically a simplified First Order decoder could be

generated by pointing a virtual cardiod mic located in the

sweet spot in the direction of each speaker intended for

reproduction (in a perfectly regular layout)

− Higher Order Ambisonics are needed to avoid the blurry

source effect a First Order Ambisonics decoder is limited by

(with limited number of microphones in a spherical sound

field), and HOA also increases the listening sweet spot where

the sound field is realistically reproduced

Rendering

− HOA are directional component added to the First Order B-format. It means every speaker contribute to any

sound in any direction equally (an isotropic sound field), which also improves localization; particularly to the

sides and rear


Ambisonics Audio

• Spherical Harmonical Functions

− When considering a sound field in space, it can be approximated by spherical

harmonical functions; and in this sense the First Order Ambisonics is the order 0

and order 1 in a Fourier-Bessel series. Naturally the approximation is only so good,

and higher orders are needed to make the approximation closer to reality.

− “A multi-channel {32-ch} … Eigen-mic array is able to truthfully measure up to 25

spherical harmonic signals of order 4” [Angelo Farina; Acoustics professor in

Parma]. The upper frequency resolution is determined by the size of the eigen-mic

and the distance between the microphones.

− Qualcomm has built a 2” spherical microphone made up by 32 microphones equally

spaced across the surface and has started using it to measure sound fields with

higher accuracy. The detailed accuracy is under evaluation.

Higher order Ambisonics


Spherical Harmonics

Compact representation of sound pressure field

Scalability: increasing HOA order N increasing spatial accuracy

N=6 49 (~50Mbps) audio signals efficient compression needed

Order

N

Number of HOA audio signals

2D 3D

0 1 1

1 3 4

2 5 9

3 7 16

4 9 25

5 11 36

6 13 49

N 2N+1 (N+1)2


Principles of Higher Order Ambisonics (HOA) (I/II)Physical description of sound pressure as a function of space and time

p1

p3

p2

HOA (N+1)2 Coefficient Signals (N+1)2 Spherical Harmonics

∞ N

Spherical Bessel

functions


Principles of Higher Order Ambisonics (HOA) (II/II)Scene Based Audio

Mezzanine format

p1 p2

pM

HOA

Transform

M microphone signals

Sound field

Manipulations

(optional)

HOA

Renderer

(N+1)2 HOA coefficient signals (3D case)p1 p2

pM

Source

Receiver

Point of

view

Accurate reproduction of sound field

L loudspeaker

signals

Fle

xib

le r

en

de

rin

g

Eigen-mic

32 Mics

HOA Transform

25-ch

HOA Renderer

8-ch to 16-ch

BRIR / HRTFs

16-ch / 2-ch

Efficient

Compression


Scene Based Audio Rendering

• Very smooth and efficient process to accommodate head movements via sound field rotation

in the HOA domain

− Yaw, Pitch, Roll

• Using BRIRs or HRTFs for virtual loudspeaker directions

− No crossfading of HRTFs necessary

Virtual Reality use case

BRIR – Binaural Room Impulse Response, HRTF – Head related Transfer Function

Decoded

Coefficient signals

BinauralizationBinaural audio (for VR)

Sound field

Rotation

HOA

to

BinauralBinaural audio

HRTF-

BRIR

Head orientation


Flexible Scene rendering

• Important to render sounds from below

− Direct sounds

− Floor reflections

• No placement constrains for virtual speakers

− More flexibility / better placement than classic configurations possible

− HOA naturally leads to t-design virtual speaker positions

SBA can be rendered with the same complexity to various loudspeaker configurations

SBA – Scene Based Audio, LFE – Low Frequency Effects

7.1+4

22.2

t-design


Scene Rendering

• Pros:

− Most VR Headsets support stereo output

− Portable, easy setup, generally inexpensive

− Listener “always” in sweet-spot regardless if recording made binaurally or with HoA Ambisonics

− Does not disturb the neighbors

• Cons:

− Shape of pinna of no help, vertical localization not working well

− Sound field with background music and speech mixes, less immersive

− Possible fatigue after extended periods of use

Binaural Audio Reproduction


Scene Rendering

• Pros:

− Common in Home environment, multi-channel receivers

− Hardware capable of heavy processing (gaming console, computer)

− Configuration supports multiple different layouts and configurations

− Less fatigue experienced as not worn on body

• Cons:

− Stationary, not a portable solution

− Listener may not be in the sweet spot; home systems not always set up well

− Walls/floor can create reflections/echo effects

Multi-channel Loudspeaker Reproduction


Scene Rendering

• Pros:

− Good sense of immersion, sounds appear from distinct directions

− Localizes all X-Y-Z sources well generally

− Available content: 3rd order Ambisonics and above becoming common in VR and 360 media online

− Sound field is accurately captured in the sweet spot with an Eigen-mic and reproduced with a large array of

speakers

• Cons:

− Lack of precision with first order, sounds do not appear from a point-source but a sphere

− Spatialization can seem blurry if panning is not done well (sources may blend)

− High frequency content is limited in First Order Ambisonics

Conclusions


Scene-based audio is a new paradigm for 3D audioProviding key benefits and solving the major challenges of existing audio formats

MIPS = Millions of Instructions Per Second

High fidelity

• Higher order ambisonics

• The perfect representation of the 3D

audio scene

• High resolution and increased sweet

spot

Efficient

• Reduced bandwidth and file size

• Rendering complexity is independent

of scene complexity

• A single format

• Scalable layering

• Power efficient: high quality per MIPS

Comprehensive

• Simple, real-time capture

• Flexible rendering

• Seamless integration into audio

workflows/applications

• Advanced effects for interactivity


Quality of Experience

• Headphone stereo highly portable, less disturbance to neighbors, low cost option

• For HOA Head-tracking; every direction in the Sound field can be equally well reproduced

• Higher Order Ambisonics enable a wide range of manipulations including rotation, reflection,

movement, 3D reverb, visualization and directionally-dependent masking and equalization

• Short Motion to Latency delay and good lip synch are needed; both are achieved by HOA

• HOA / Scene based audio can be very efficiently compressed using MPEG-H which includes

spatial compression techniques

• Object based audio (also supported in MPEG-H) can be used in conjunction with Scene

based audio to add a few highly localizable/controllable sound sources if desired, eg. Non-

diegetic Voice commentary

Benefits

Motion-to-latency delay: Delay between when sounds arrive to either ear will to the listener dislocate the source, if the head is moved to intuitively improve localization this is perceived as very annoying (spatial av-synch during head motion)

Lip-synch: When the lips are seen moving vs. when the ears register the speech arriving (temporal av-synch)


Quality of Experience

• Binaural recording/reproduction lack immersive feeling

• Ambisonics suitable for immersion and envelopment; in First Order sources may blend

• Higher Order Ambisonics needed for realistic high resolution reproduction

• Audio objects have to be individually compressed and transmitted with the metadata. This

might not be feasible as the number of objects increases

Concerns


Scene-based audio is an ideal solution for VR

Capture Playback

A natural fit for capturing and playing back 3D positional audio

High fidelity

• Captures the entire 3D sound scene

in high quality

• Video and audio captured on

the same device

Real-time & simple

• Works on a variety of devices (action

camera, smartphone, etc.)

• No post-production required but easy to

apply scene-based effects

• Great for live events like sports and user-

generated content

• Compact file

SoundsSo accurate that

they are true to life

Immersive

• High-fidelity 3D surround sound adjusts

based on head pose

• 3-DOF and 6-DOF support

• A natural way to guide a user’s attention

Efficient

• Accurate manipulation of

the sound field

• HOA coefficients are computationally

efficient to rotate, stretch, or compress the

audio scene

Follow us on:

For more information, visit us at:

www.qualcomm.com/mpeg-h-scene-based-audio

Nothing in these materials is an offer to sell any of the components or devices referenced herein.

©2016 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.

Qualcomm is a trademark of Qualcomm Incorporated, registered in the United States and other countries. Other products and brand names may be trademarks or

registered trademarks of their respective owners.

References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsi diaries

or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast

majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries,

substantially all of Qualcomm’s engineering, research and development functions, and substantially all

of its product and services businesses, including its semiconductor business, QCT.

Thank you

@2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. 26@2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies.

All Rights Reserved.

Documents

ETSI STQ: Workshop May 2017 Multichannel VR Audio ... · PDF fileMultichannel VR Audio Rendering to Stereo; Technique and Quality concerns ... (action camera, smartphone, ... •Great