Principles and applications of 3D reproduction using...

Principles and applications of 3D reproduction using Ambisonics

Alois Sontacchi

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi 2

@ the focus of Art and Technology

Institute of Electronic Music and Acoustics

University of Music and Performing Arts Graz

AUSTRIA - 8010 Graz - Inffeldgasse 10/3

3

artistic/scientific technical/scientific

Professors 1 1

Academic non-professorial teaching staff (scientists) 6 12

Academic non-professorial teaching staff (lectors) 2 7

Other staff members5

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

4

Education

• Musicology (Bachelor, Master) in cooperation with KFU Graz

• Composition - Computer Music (Master + NEW: Bachelor)

• Electrical and Audio Engineering (Bachelor, Master) in cooperation with TU Graz

• Doctoral school

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

5

Artistic Research

sound & spaceembodimentalgorithmic composition

Signal Processing and Acoustics

spatial audioaudio signal processingpsychoacoustics

Computer Music

sonificationinteraction design

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Research Areas Sound & Music Computing

6

Motivation

What is the necessity for 3D Audio production ?

Games & Entertainment

Music Performance / Composition

Listening Test: In-Situ Simulation- speech intelligibility in noise

- evaluation of hearing-aids

- improvements of ANC-applications

Immersive Telepresence

LoRA A Loudspeaker-Based Room Auralization SystemS. Favrot, J. M. Buchholz, Acta Acoustica Vol. 96, p.364-375, 2010.

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

7

Outline

Introduction

Ambisonics Theory

Optimization criteria for reproduction quality

Applications

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

8

Psychologists/psychophysicists discover „phantom source“ around 1900

- „phantom“ sound localised, where there‘s nothing physicalmedical term such as „phantom pain“

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

9

X/Y uses first-order transduer characteristics to produce psychoacoustic effect

Introduction

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

10

X/Y uses first-order transduer characteristics to produce psychoacoustic effect

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

11

X/Y uses first-order transduer characteristics to produce psychoacoustic effect

-15dB

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

12

M/S is useful for modifying coincidence stereophony image

- + +

+- ++Encoding

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

13

M/S is useful for modifying coincidence stereophony image

- + +

+- ++Encoding

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

14

M/S is useful for modifying coincidence stereophony image

- + +

+- ++

- + +

+-++

But don‘t increase S: out-of-phase

Encoding

Decoding

Introduction

Advancements of Ambisonics

15

Soundfield microphone

A equilateral triangle is a regular 2D simplex; for 3D: tetrahedron

- +-

+

+

+ + - -

FLD FRU

BLU BRD

Front/Back,Left/Right,Up/Down

+

y

FRU

BRD

FLD

BLUxy

z

+-

x z w

+ - - + + + +-+ - +

Encoding

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

16

Decoding w by sampling

Simplistically

-+

- ++

+

+

+

+

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

17

Decoding x by sampling

Simplistically

-+

- ++

- +

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

18

Simplistically

-+

- ++

-

+

Decoding y by sampling

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

19

Matrix(Decoder)

Lspk. setup

The entire encoding and decoding signal chain

Matrix(Encoder)

Mic. arrayconfig. Andproperties

Ambisonics format

instead of array recording, monosignals can be encoded by vector.

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

20

Matrix(Decoder)

Lspk. setup

The entire encoding and decoding signal chain

Matrix(Encoder)

Mic. arrayconfig. Andproperties

instead of array recording, monosignals can be encoded by vector.

Experimental evidence: what is heard ?

How to arrange loudspeakers optimally?

Ambisonics format

How to best shape the decoder?

Is it sound field synthesis? What‘s its theory?

Higher-order recording unclearbefore 2000… It won‘t workanymore using 2D/3D directionvectors

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Introduction

The theory behind sound field synthesis lies in theKirchhoff-Helmholtz integral (KHI) equation

21

Theory of sound field synthesis, see Schenk (CHIEF point method) or Burton-Miller solution method

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

22

d/dn p|Γ is the surface normal derivative of sound pressure(proportional to normal component of particle velocity)

Theory of sound field synthesis, see Schenk (CHIEF point method) or Burton-Miller solution method

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

The theory behind sound field synthesis lies in the KHI equation and equivalent scattering problem

23

For simplicity (no encoding of source distance):Considering amplitude panning, the driving function onlyselects active sources on the boundary.

g is a virtual panning function: beause it is continuous(loudspeakers will be discrete sources)

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

The theory behind sound field synthesis lies in the KHI equation and equivalent scattering problem

24

Virtual panning function (with height)

red>0

blue<0

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

enc

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

25

Virtual panning function (with height)

red>0

blue<0

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

enc

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

26

Virtual panning function (with height)

red>0

blue<0

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

Advancements of Ambisonics

27

Virtual panning function (with height)

red>0

blue<0

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

How to optimally arrange withloudspeakers?

Virtual panning function must bedecoded to discrete loudspeakers

How can they represent it well?

Advancements of Ambisonics

Trying to do sound field synthesis seriously, with spherical functions:

Panning and decoding [Zotter, Pomberger, Frank, „An Alternative Ambisonic Formulation“, 126th AES Conv, May, 2009.]

29

“psychoacoustic” measures were considered for decoding.

Based on theoretical consideration

f<1.5kHz

loudness direction of localisation

f>1.5kHz

loudness direction of localisation

Makita theory.

What is relevant for broad-band sounds? What can we demand?

Trying it more psychoacoustically:Gerzons measures for the decoding problem

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Jérôme Daniel‘s shapes for virtual panning functions: which is best?

Insertion of order weights an

basic

Max-rE

in-phase (a bit more complicated…)

30

Panning and decoding [Jérôme Daniel, PhD thesis, 2000]

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

31

loudness direction of localisation

loudness direction of localisation

Gerzons measures applied on virtual panning function

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

linear measures

quadratic measures

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

32

Virtual panning functions have nice properties:

panning-independent properties of Ambisonic panning functions:

• symmetric around panning direction ϑs

• amplitude integral is constant (V)

• directional amplitude concentration:- constant magnitude of concentration ||rV||

- always points towards panning direction ϑs=rV/||rV||

• energy integral is constant (E)

• directional energy concentration:- constant magnitude of concentration ||rE||

- always points towards panning direction ϑs=rE/||rE||

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

33

Nice properties might get lost after decoding:

• symmetric around panning direction ϑs

• amplitude integral is constant (V)

• directional amplitude concentration:- constant magnitude of concentration ||rV||

- always points towards panning direction ϑs=rV/||rV||

• energy integral is constant (E)

• directional energy concentration:- constant magnitude of concentration ||rE||

- always points towards panning direction ϑs=rE/||rE||

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

Which are relevant? Psychoacoustically?

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Vector rE is most useful measuring localisation direction

34

broad-band sounds

VBAPVBIP

horizontal-only level panning

Psychoacoustically relevant factors [Matthias Frank… not yet published.]

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Which sampling preserves all energetic measures?

35

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

36

Virtual t-design Ambisonics using VBAP

Fig. 6: Triangulation, energy measure, and spread measure as a function of the virtual source direction. [Zotter, Frank, Sontacchi, EAA 2010] [Zotter, Frank, JAES 2012],

PhD-Thesis Matthias Frank, 2013

VBAP

Virtualt-dsgn.Ambi.N = 5

00.510

10

20

30

40

50

60

70

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

37

VBAP and Ambisonics compared

Triplet-wise panning (VBAP)+ constant loudness+ arbitrary layout-- varying spread

Ambisonic Panning~+ constant loudness+ arbitrary layout~+ invariant spread

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

Principles and applications of 3D reproduction using Ambisonics Alois Sontacchi

Psychoacousticians are sceptical aboutAmbisonics with height.

• Two vertically stacked loudspeakers do not yield a stablephantom source; should not be more than 30° separated

• A loudspeaker triplet does not yield stable phantom sources in all orientations, and results are individual

• However: What if we do not understand Ambisonics as a simple pair ortriplet?How diffuse must it be to obtain stable playback?

• [Results Braun/Frank/Pomberger 2011]

Advancements of Ambisonics 38

39

[Sabastian Braun, Matthias Frank, „Localization of 3D Ambisonic Recordings …“ ICSA, 2011]

Advancements of Ambisonics

40

[Sabastian Braun, Matthias Frank, „Localization of 3D Ambisonic Recordings …“ ICSA, 2011]

Advancements of Ambisonics

Application for Loudspeaker Reproduction

41Advancements of Ambisonics

1 2

Ambisonics Decoder

Encoderposition Encoder position

Ambi-sonicsBus

3 4

Encoderposition Encoder position

AmbisonicsRecording

rotation

Live electronics in modern operas with Ambisonics:

Alois Sontacchi 2002Thomas Musil

Markus Noisternig 2004Thomas Musil

Peter Plessas 2010

Pomberger, Zotter: An Ambisonics Format for flexible playback layouts, 1st Int. Ambisoncis Symposium, 2009.

Application for Headphone Reproduction

42Advancements of Ambisonics

Sources Encoding Room Head Rotation Filter Reproduction

43

I would particularly like to thank

Franz Zotter

Matthias Frank

Hannes Pomberger

Advancements of Ambisonics

44

Available software tools

• Tools:

CUBEmixer (IEM)

spat (ircam)

AmbDec, Amb plugins (Fons Adriaensen)

ICST Max/MSP plugins

plogue bidule

• Soon more to come.

• If you are a mixing desk manufacturer…

Advancements of Ambisonics

45

Format

• Format

• International Conference on Ambisonics and Spherical Acousticshttp://www.aes-uk.org/uk-conference/25th-call-for-papers/

Advancements of Ambisonics

46

How to record higher order? – upscaling methods from first order

- DirAC: Fraunhofer / V. Pulkki

builds intensity vector from p, vx, vy, vz at each frequencyto assign narrow direction

determines diffuseness

- HARPEX: S. Berge

can isolate even two signal directions at each frequency

Panning and decoding [Franz Zotter, Matthias Frank, „All-Round Ambisonic Panning and Decoding“, JAES, October 2012]

Advancements of Ambisonics

47

Meyer, Elko 2000-

Petersen, Hald, Moller-Juhl, B&K, 2003-

Jin, v.Schaik, 2006-

Li, Duraiswami, O‘Donavan, Grassi, visisonics, 2004-

Rafaely&Park, 2004-

Higher order microphone arrays

Takashima, Nakagawa,Williams, 01db, 2008

Advancements of Ambisonics

Pausch, ZotterPomberger, 2013

40 years of coincidence recording yield Ambisonics

• 1930s- Blumlein‘s coincidentdirectional microphone pair (X-Y, M/S)

- Olson‘s ribbon microphones (figure-of-eight)

• 1940s- microphones with acoustically switchable characteristics

- Bauer‘s supercardioid has maximal front/back ratio

• 1950s- Neumann‘s U57 and AKG‘s C12 (double membrane) studio microphones allow electronic switching

- AKG‘s logo

- Vinyl and tape records available in stereo

- (many non-coincident A/B)

• 1960s- BBC broadcasts in stereo

- Psychoacoustics on stereo

• 1970s- Quadraphony

- Matrix stereophony: encoding/decoding in azimuthalharmonics (Cooper, Shiga)

- Ambisonics(Gerzon/Fellgett/Craven): recording: CALREC-triplet, Soundfield microphone, decoding/encoding circuits

- Panning in higher orders

Time-line

40 years later: a hand-full of higher orders, enriched by sound field synthesis theory, psychoacoustic experiments, and a community

• 1970s to 1990s: - theoretical psychoacoustic decoding, electronics (Gerzon) V,E,rV,rE

• 1990s: - binaural Ambisonics (Jot)- Sursound list- Pulkki‘s papers and experiments about 3D stereophony/VBAP

• 2000s: - Daniel (300 pages: theory), - Ward/Abhayapala truncation error- Malham (energetic distribution), - Sontacchi/Höldrich/Noisternig/Zmölnig(IEM-Bin-Ambi, IEM-CUBE

- Poletti („Mode-matching“),

• More in 2000s: - Meyer/Elko(„Eigenmike“, 4th-order recording)

• End of 2000s:- experimental psychoacosutics: Bertet, Solvang, Kratschmer, Frank, Bates, Marentakis

- Advancement in theory(concepts, sampling, truncation): Ahrens/Spors, Fazi, Li, Rafaely

- First Ambisonics Symposium: HRTFs, musical instrumentradiation, embedding of Ambiinto scene description, …

- Zotter/Pomberger/Frank …

Time-line