Advanced real-time sound and data processing with FTM&Coimtr.ircam.fr/imtr/images/Ftm_workshop_STEIM_2010_slides.pdf · 4. FTM & Co — IMTR Team — IRCAM - Centre Pompidou FTM Motivations

Advanced real-time sound and data

processing with FTM&Co STEIM Workshop 10–13.5.2010

presented by Diemo Schwarz

IMTR Real-Time Music Interaction TeamIrcam–Centre Pompidou

http://ftm.ircam.frFTM 2.5.0 beta release (BETA 14) for

Max 5 on Mac OS X

Wifi: Studio 3PW: AACCBBDDAA

1

http://www.ircam.fr/equipes/temps-reel/maxmsp


http://ftm.ircam.fr/downloads/FTM.2.5.0.BETA.14-Max5.dmg








FTM & Co — IMTR Team — IRCAM - Centre Pompidou

PlanFTM

Introduction

Basics

Examples

Persistency: SDIF, SQLite

GaborPrinciple

Examples

MnMGesture Follower

CataRT

2


What is FTM&Co?a real time object system for Max/MSPrepresentation of musical structuresrepresentation of sound data and structuresabstraction from the real-time system (Max, PureData, ...)Timeline:

Reference article:Norbert Schnell, R. Borghesi, D. Schwarz, F. Bevilacqua, R. Müller « FTM — Complex data structures for Max », International Computer Music Conference (ICMC), Barcelona, 2005

— Max/ISPW — Max/FTS — jMax — FTM — t

3

Introduction

courtesy of Mikhail Malt

4


FTM MotivationsAudio analysis/synthesis (Gabor)

representation of grains, waves/periods and framesrepresentation and organisation of sound descriptors

Gesture analysis, classification and recognition (MnM)(linear) algebra and matrix processingrepresentation and (statistical) modelling of gestures gestalts

Score representation and following (FTM, suivi)representation of a score or time series of datarepresentation of objects in a score

➡ modularity & flexible structuring of memory

5


FTM ComponentsFTMlib shared library for Max

message system extensionintroducing complex data types

+Max externals

+Editors and graphical externals

6


FTMlib FeaturesObjects of predefined classes

mat, dict, track, fmat, expr, bpf, ...

Expression evaluationFile import/export (soundfiles, MIDI, SDIF)Object serialization and persistence

7


TerminologyReal World Object-Oriented

Programming FTM Max

a thing with a name, state, and properties (an alert black-and-

white dog named Otto)

object (data) object (some specific Max objects = externals)

an action and reaction(“sit!”)

method and return value

message andreturn value

message(and output)

a process (the vet) (function, process) module

(external)Max object (external)

a class of similar things (dogs)

class class –

a more general or more specific class (pug dogs, animals)

superclass, derived class

– (types, related classes) –

8


FTM Conceptsseparation of data and operators

➡ data objects of optimised classes with methodsfile import & export (text, audio files, MIDI files, SDIF)

persistence (save within patcher and in separate files)

conversion to Max values and listssent within Max values and lists (as references)

simple garbage collector (by reference count)

➡ operators

9


The FTM Package Familiy

Gesture Follower CataRT

Gaboratomic sound

MnMmaths+stats

Suiviscore

following

FTM externals

FTMdata classes, visualisation, editing

SDIF, SQLite

application packages

basic library

http://ftm.ircam.frhttp://imtr.ircam.fr

applications

10

http://www.ircam.fr/ftm

http://www.ircam.fr/ftm

http://freesoftware.ircam.fr/wiki

http://freesoftware.ircam.fr/wiki


DistributionFTM 2.5 current stabilizing versions for Max5

FTM 2.3 and 2.4 for Max 4.6

FTM is free (libre) open source software (LGPL)Gabor&MnM are free (as in free beer)

both are distributed together

Web site http://ftm.ircam.frUser mailing list [email protected] on http://lists.ircam.frDeveloper list [email protected]

11




Credits and ReferencesFTM&Co is developed in the Real-Time Musical Interactions team at IRCAM by Norbert Schnell and Riccardo Borghesi, Diemo Schwarz, Frederic Bevilacqua, Remy Müller, Jean-Philippe Lambertwith contributions by Julien Bloit, Baptiste Caramiaux, Arshia Cont, Pierre Duquesne, Sebastien Gulluni, Baptiste de la Gorce, Bruno Zamborlin

Reference article:Norbert Schnell, R. Borghesi, D. Schwarz, F. Bevilacqua, R. Müller « FTM — Complex data structures for Max », International Computer Music Conference (ICMC), Barcelona, 2005

12

http://imtr.ircam.fr/





G a b o r :Dennis Gabor, Acoustical Quanta and the Theory of Hearing, Nature, 1947

Multi-Representation grains, waveforms, spectral frames, partial sets

Real-Time extension modules for Max/MSP, PD

Analysis–Synthesis...and transformation in a unified framework

Reference article:Norbert Schnell et al. « Gabor, Multi-Representation Real-Time Analysis/Synthesis », COST-G6 Conference on Digital Audio Effects (DAFx), Madrid, 2005

13


Two Fundamental Ideas...1. Atomic sound particles in various descriptions

time-domain: grains, wave periodsspectral domain: FFT frames, phase vocoder frames, sinusoidal partialsmetadata: cepstrum spectral envelopes, filter coefficients, sound descriptors

2. Arbitrary rate processingpicking grainspitch synchronousmultiple frame rates

14


Overview

15


...their Realisation...1. Unified framework

sound particles represented as generic data objectsmatrix of floating point numbers

streams as time-sequence of objects

(others: hash-table or matrix of objects)

sent in between specific analysis or synthesis modules

2. Sound processing via message passing (control)64 bit time tags for sub-sample precise reconstitution of the signal input/output buffers with interpolation

16


...and their Advantages1. Unified framework allows generic

manipulation, storage, and display of atomshigher modularitycan use mathno longer needing a monolithic analysis+synthesis+manipulation module

2. Arbitrary control rate processingis adaptable to frequency/texture/rhythm of sound or analysis algorithmnot limited to fixed-rate audio DSP chain

17


From Signal to AtomWays to cut the input buffer into vectors:

arbitrary grain extraction: gbr.grain~

pitch synchronous elementary waveforms: gbr.psy~based on yin pitch estimation algorithm

(de Cheveigné and Kawahara, JASA 2002)

constant rate frames: gbr.slice~FFT

phase vocoder analysis

additive analysis

18


From Atom to SignalWindowingInverse FFTFFT-1 (additive synthesis)Overlap add: gbr.ola~

shortcut for paste and drain (output delay line buffer)

19


...and in between...Manipulation

of the dataof the timing

DisplayStorage and retrieval

files (audio, SDIF)databases

Future: network or inter-application send/receive via OSC

20


Gabor Summary

Ideas Realisation

atomic sound particles in multiple representations

unified framework of specific analysis–

synthesis, generic data

arbitrary rate processingevent processing with time-tags + delay lines

21


MnM package

Classification and recognitionmotion capture data, sound and music

Basic matrix calculationsLinear algebraPCA, GMM, HMM, NMD, EM, etc.

Reference article:Frederic Bevilacqua, R. Muller, N. Schnell « MnM: a Max/MSP mapping toolbox », New Interfaces for Musical Expression, Vancouver, 2005

“Mapping is not Music”

22


MubuPackaging of frequently used signal analysis, representation and synthesis algorithms with perfect synchronization as MSP externalsMailing list [email protected] article:Norbert Schnell, et al. R. Borghesi, D. Schwarz, F. Bevilacqua, R. Müller « Mubu & Friends - Assembling Tools for Content Based Real-Time Interactive Audio Processing in Max/MSP », International Computer Music Conference (ICMC), Montreal, 2009

23

Real-Time

Corpus-Based

Concatenative Synthesis:

CataRT

24

Diemo Schwarz, Etienne Brunet– Synthèse sonore concaténative par corpus en temps-réel

IntroductionReal-time Synthesis interactive + explorative (navigation)concatenative grains + overlap + chainingcorpus-basedsounds + segmentation + descriptors (timbre, metadata, classes)

25

Diemo Schwarz – Real-Time Data-Driven Sound Synthesis

Principle of CBCSLarge sound database

Segmented into units

si la do

si do dore

la do domi sol

si la do

■ synthesis target

■ unit selection from the sound database to best match the target

■ Synthesis by concatenation of the units

26

Motivation and Approachwork with all the nuances of real sound

large sound databases exist, ready to use

new method → new sound creation

data-driven vs. rule-based approach

general superiority of data-driven approaches in many other domains

27

Influences and LinksInspired by concatenative speech synthesis

CBCS in real-time (as implemented in CataRT) can be seen as a content-aware extension to granular synthesis:

providing direct access to specific sound characteristic

not restricted to selection by position in a sound

28


Applications of CBCS

Interactive exploration of sound databases (browsing)

Resynthesis of an audio target (mosaicing)

Texture synthesis (ambiences, soundtracks)

High-level instrument synthesis (example: Synful)

Expressive speech synthesis

29

A Short History of Data-Driven SynthesisI.Prehistory (1948)

Concrete and early electronic music

II.The Classics — revisited (1980, 1990)Sampling

Granular Synthesis

John Oswald

III.The Modern Times (2000)many research projects, often in mutual ignorance

30

Tape splicing

soft attack or decay

combined attack and decay of two sounds

medium attack or decay

hard attack or abrupt finish

softer and less abrupt than D

[figures: Mikhail Malt]

Karlheinz Stockhausen: Étude des1000 collants (1952)

31

John Cage: Williams Mix (1953)

[http://www.medienkunstnetz.de/works/williams-mix]

32

http://www.medienkunstnetz.de/works/williams-mix

http://www.medienkunstnetz.de/works/williams-mix

Level of “Data-Drivenness”

1948

19801990

1993

20002001

2001

20012002

2003

2003

2003

2003

2003

20042004

2004

20042005

2005

2005

Selection

Let them sing it for you

Granular

Samplers

Synful

Musescape

CataRT

MoSievius

stochastic SoundscapesSoundclustering

La legende des sieclesN.A.G.

MATConcat

targeted Directed Soundtracks

Soundmosaicing

Input driven resynthesisMPEG!7 Audio Mosaics

Soundspotter

automatic

Ringomatic

Musical Mosaicing

Caterpillar

Audio Analogies

manual segmental high!level

Musique Concretemanual

Plunderphonics

similarity descriptorssimilarityframe spectrum

Analysis

fixed

2005

2003

[Schwarz, JNMR 2006]

33


CataRT Introductionflexible testbed to experiment continuation and interaction: CataRT

realisation of CBCS in real-timepatch for Max/MSP with FTM, Gabor, MnM by Norbert Schnell et al. from http://ftm.ircam.fr

released as Free Software under GPL at http://imtr.ircam.fr

modular MVC architecture:building blocks for you to use

34

CataRT Overview

CorpusAnalysis

DB

Synthesis

Audio Input

35

Analysis

Corpus

Soundfile+markers

+descriptors

Persistent Sound and Unit

Database

descriptor analysis

Soundfile+grain sizeor markers

RT audio input

Audio Input

onset detection

Soundfile

36

Diemo Schwarz, Etienne Brunet– Synthèse sonore concaténative par corpus en temps-réel

Synthesis

Corpusnearest

neighbour selection

Target descriptor values Control

transformation + concatenation

descriptor analysis

RT audio input

onset detection

controller

37

Diemo Schwarz – CataRT workshop

Architectureone catart.data per corpus

one catart.import for segmentation and analysis

any number of catart.data.proxy to access data

any number of catart.lcd for display and control

any number of catart.selection per corpus

any number of catart.synthesis per selection or per corpus

38

Composers and Projects using CataRT

Sound design and compositionLuis Naõn, Hans Tutschku, Matthew Burtner, Sebastien Roux, Hector Parra, Roque Rivas, Bruno Ruviaro

InstallationPlumage (project Enigmes about the navigable score, Roland Cahen), Pierre Jodlowski, Cécile Babiole

Live electro-acoustic music Dai Fujikura, Stefano Gervasoni, Sam Britton, Aaron Einbond, Sebastien Gaxie, Miguel Angel Ortiz-Pérez, theconcatenator

Examples: myspace.com/catartsoftware

!"

!"

18

""

""

q = 60

q = 60

!"

!"

q = 102

q = 102

!#

!#

""

""

!"

!"

A. Fl.(smallgroup)

B. Cl.(smallgroup)

Tpt.(smallgroup)

1 Vln.(smallgroup)

1 Vc.(smallgroup)

electronics

Fl.

Ob. I

Ob. II

Cl. I

Cl. II

Bsn. I

Bsn. II

Hn. I

Hn. II

Hn. III

Hn. IV

Tpt.

Tbn. I

Tbn. II

Timp.

B. D.

Vln. I

Vln. II

Vla.

Vc.

Db.

!

"

!

"

!

"

!

"

!

"

!

"

!"

$SMALL GROUP

%&'%( % % %) % # % #

p

* + + ,ca. 20 - 21 sec.

, ,- , , , ,

$ % %( %&'(% % % #

p

%) * , , , ,- , , , ,

$ % % #

p

% #) . + , , , ,- , , , ,

$ %&'( #% % #

% % %p

+ / , , ,- , , , ,

0 $% %1 % % %p

% #) . , , , ,- , , , ,

$

section 3

*

4

%%%%%%%%%(

211( %%%%%%%

((

more and more dense and violent(grain size gets smaller and smaller)Harmony 3Harmony 2

gliss.

gliss.

+3 semitones,

-4 semitones,

%%%%%%%%%1

2222 %%%%%%%22(1 ,

CUT OFF

5

34 5 . * + /

$ , , , , ,- +marcato

*ff

$%'6 $%

'6$%

'6%7( %7($%

'6

%7 . . %7 . $%'

( 6$%

'6%7( %7 .$%

'6 %78 . * $%'6%78 . * .

$%'

6$%

'( 6

%7 .%7

*sffffz

$%'6%7'( % * * espress.

fff

. $%'6 %8 %( %

$%'6

%% #

$ , , , , ,- +marcato

*ff

$%'( 6 $%'

( 6%7 %71 * $%

'6 %78 . + $%'( 6 $%'6

%7 .%7(

.$%

'6$%'6%7 %7 * %

%')$%

'( 6

$%'6

%7 %7

. %7( *sffffz

%7'(8 . *

espress.

*fff

%) $%'6% %( # % %

$ , , , , ,- + marcato*ff

. %%'()

$%'6

%74 $%'6%7(

. . %7 .

$%'

( 6 %78 *$%

'6 %7(8 . * * $%'6

$%'6

%7 %7 . $%

'( 6 %7(8 * .

$%'6

sffffz

%78fff

%7'(. . % %8 . * + *

espress.

%6

$ , , , , ,- +marcato

*ff

%%') %74 .

%%

'

2 )%%')

$%'6

%7%7 .

%7

(.

%%')

%%

'

)%%

'

)%7( %7 %7* $%'( 6 $%

'6

%%

'

)

%7 %7(%%')

%%

'

)%7. %7 %7

. %%')

%%

'

)$%

'6

%7

( .%7( %7 .

sffffz fff

$%

'

( 6

%7'

(

%(. + *

espress.

%%')

%%(

%%'

()%

%

$ORCHESTRA

, , , , ,- +marcato

* .ff

%%')

$%'( 6

%74%%

'

2 )%7 .

%7. .

%%

'

) %%')%7 %7 . %

%')%%

'

) %%')

$%'

( 6

%7 %72. %7

$%'

( 6

%%')

%7( . . %71%%

'

)%72. %7

.%%

'

)%%')

sffffz

%7 %7fff

espress.

%%'

()$

%

'

(6

%7'% %( %

% #%

% #%

%%

'

)%

% %

9 , , , , ,- +marcato

*ff

$%'2 6%78 .$%

'6

$%'6%7 . %7 .

$%'

6$%

'6$%'2 6 $%

'6%7 %72 . %7 $%'6 $%'6

%%')

%7 %7 . %7%%')

$%'6

%7 . . %7$%

'6%7 .%7

.$%

'6 %78 .

sffffz fff

$%'

( 6 espress.

%7'

2%

%( . $%'

( 6%( %( % %) *

$%'6 % # %

9 , , , , ,- +marcato

*ff

%%') %74 . .

$%'6

$%'6%7

%7 . %

%'2) %%') $%

'6%7 %7 . %7 * %

%') %%')

$%'6

%7 %7$%

'6%7 . %7 . . %%') %74 .

$%'

6 %78 *sffffz

espress.

fff

%7' % %( % %

) * $%'

( 6 %( # % %) *

$ , , , , ,- + + marcato

ff

. $%'6

%74 * 9 . $%'6

$%'

2 6%7. %7 . $%

'6 $%'6%7 . %7 * . $%'6%78 * $%

'6 %78 . *sffffz

ff

$%'

( 6%7' * %7( * $%'6 %7(8 . + .$%'6 %78 *

$ , , , , ,- + 9+ marcato*ff

$%'2 6 %78 . .$%

'6 %728 *$%

'6%78 . *$%

'6%78 . * $%'6 %78 . * $%

'6 %728 .sffffz ff

$%'6%7'

. . %( . $%'( 6 %78 * $%'6

$%'

( 6%7 . . %7 + $

$ , , , , ,- + marcato9 *ff

$%'6%78 . .

$%'6%78 * * . $%

'6$%

'6%728 $%'6%7 . . %7 + * $%'

( 6 %78 . * $sffffz

%7'8 . *ff

*$%

'6 %7( 4 . * *

$%'

6%74 . * 9

$ , , , , ,- + 9*ff

marcato$%'2 6%78 . * $%'6%78 . . $%

'6%74 * * . %

%') %74 * . $%'6

$%'6

%74 %728 . * .$%

'6

sffffz

%78ff

%%')%7'( %7( * * $%

'( 6 %78 . * $ . $%'6 %7( 4 *

%%'

() %7( 4 .

$ , , , , ,- + + marcato

ff

$%'( 6%78 . * *$%

'6 %78 . .

$%

'6 %%

')%72

. %7. $%'( 6%78 * . $%

'6 %7(8 * *sffffz ff

$%'6%7' %7 * * . $%'6 %78 + .$%'6 %78 *

9 , , , , ,- , , ,(senza sord.)

sffffz

%7'2 4 . * + /

9 , , , , ,- , , ,straight mute

sffffz

%7'(8 . *super short staccato

+p

%7'( 7' 7' 7' 7'

5

gliss.

9 , , , , ,- , , ,p

soft sticks

%'( ' ' ' ' ' ' ' ' ' ' '3 3 3 3

gliss.

: , , , , ,- , , ,sffffz

%7'4 . * + /

$ , , , , ,- marcato+ *ff

$%'6 %7(8 .

$%'6

$%'

6%7( . . %7 * $%

'6 %7(8 .$%

'6 $%'6%7(

. .%7

* $%'6

%74 . . $%'

( 6 %7(8 *$%

'6 %74 .

sffffz

unis.

fff

$%'6 espress.%7'( %

%($%'6

% %% %( % %4 . *

$ , , , , ,-div.

marcato+ * .ff

$%'6 %7(8 * $%

'6 %78 . . $%'( 6 %7(8 * *

$%'6 %78 .

$%'6 $%

'( 6%7 . . %71 . $%

'( 6 $%

'6%%')%7(

.%7

. %7

$ , , , , ,-

div.

marcato

+ *ff

$%'( 6%78 .$%

'6$%

'6%7(. .

%7(*

$%

'6 %74 . %

%'()%%')

%7

( . . %7* . %

%') %78 .$%

'( 6

$%'

6sffffz

unis.%7* .

%7

2fff

$%'6 espress.

$%'

6%7'( %7 . %(

%) * /

$ , , , , ,-marcato

+ff

* .$%

'( 6

%718 * . $%'6 %7(8 .

$%'

( 6 %78 *

$%'6

$%'6%7( . %7 .

$%'6

$%'

6%7

. %7 .

$%'

6 %%')%72

. . %7

( *

0 , , , , ,-div.

marcato+ *ff

$%6%74 . $%

'2 6 %

%')%7 . . %7 * $%

'6 %78 . . $%'6 %72 4 *

$%'

( 6$%

'6%7 . %7 . %

%') %78 . $%'

2 6%74 .

$%'

6 %74 .

sffffz

unis.

fff

espress.%7' %( %$%

'6%

%( % %% #

% % *

0 , , , , ,- marcato+ff

* .$%( 6

%78 * $%'

2 6%74 . . $%'6 %72 4 *

$%'

( 6 %78 . *$%

'6$%

'6%7 . %7 . . $%

'6%78 * . %%') %72 4

9 , , , , ,- +marcato

*ff

$%'6

%74 . .$%

'6 %78 * $%'2 6$%

'6%7 . . %7 * $%

'6%74 . $%'6

%%')%7 . . %7 * .

$%'

6 %78 *

sffffz

unis.

%%'()

espress.

%7'%( .

fff

. %%')

% %($%

'( 6

% % % +

9 , , , , ,-div.

+marcato

*ff

.$%

'6%78 * $%

'6%74 . . $%

'6 %78 *$%

'6

$%'6%7

. %7 . $%'2 6

%%')%7 . . %7 +

$%'

6 %78 .

9 , , , , ,- +marcato

*ff

$%'6

%78 . $%'6$%

'6%7. %7 . .

$%'6

%78 * .$%

'2 6 %78 *

$%'6

%78 . * .$%

'6%78 +

sffffz

%7'%7( * *

fff

$%'6 %(

) + * %)

2

39

Installation GrainStick“Rainstick” gestural control metaphor

Demo of SAME (Sound And Music for Everyone Everyday Everywhere Every way) EU project

Composition of sounds and ambiences by Pierre Jodlowski

WFS spatialisation and Motion Tracking by Grace Leslie and the Room Acoustics team

Gesture analysis by Bruno Zamborlin and Diemo Schwarz

40

Installation Xe-RocksTurning photocopiers into musical instruments, by video and sound artist Cécile Babiole

On show at the ECM Gantner, Bourogne

41

Current research and applications at the Real-Time Music Interaction Team (IMTR) — http://imtr.ircam.fr — Diemo Schwarz

Installation PlumageCollaboration with national design school ENSCI led by Roland Cahen, Yoan Ollivier, Benjamin Wulf (ENSCI), Christian Jacquemin, Rami Ajaj (LIMSI), Diemo Schwarz (Ircam)

within the project ENIGMES about the navigable score3D interface using Virtual Choreographer http://virchor.sourceforge.net

42


Plumage

43


Plumage

44


Plumage

45


Musical Applications (1)Corpus-based improvisation with live sound input

Diemo Schwarz with Etienne Brunet

Live Algorithms for Music (LAM) concerts:Improvisation of George Lewis, Evan Parker, Sam Britton, Diemo Schwarz and othersRien du tout by Sam Britton, Diemo Schwarz

Rearranging, navigation, live-recording, re-orchestration

46


47


Musical Applications (3)Whisper Not by Stefano Gervasoni for viola and electronics, computer music realization by Thomas Goepfer

“response” of CataRT to the instrumentmorphing between corpora of pizzicato viola and water-drops

Interaction with pre-recorded sound,

interpolation, rearranging, navigation

48


Musical Applications (4)swarming essence by Dai Fujikura for orchestra and electronics, computer music realization by Manuel Poletti

harmonic selection of specially composed small instrument group and extended playing stylecorpus-based orchestration, separate treatment

Re-orchestration, rearranging, navigation

49

Current research and applications at the Real-Time Music Interaction Team (IMTR) — http://imtr.ircam.fr — Diemo Schwarz

Musical Applications (5)Beside Oneself, Aaron Einbond, 2008 for viola and electronics:

real time control of CataRT from audio analysis of the viola

What the blind see 2009 for small ensemble and electronics

real time control of CataRT from audio analysis of the viola

corpus-based “orchestration” or transcription

texture example:1. Rain on leaves

2. CataRT resynthesis with extended technique instrument sounds

50

Aaron Einbond – What the blind see

B

&

&

ã

ã

t

&

ã

Alto

Cl.B.en Sib

Hp.

Pno.

Vib.

F

!

" " "

C10

C10

C10

a tempo

a tempo

a tempo

1936

T T .T#

Œ Ù .T

$T

%6

& . ym

bruit des clefs : doigtés ad lib.

"f "yn y

7

TmflM

ßym y ym ym y Ù TM

flßy ym y y yb

7

® x xx x xm

œSF neige to concatenation

Œ

7

‰ ® jœb œbp

œœ œ œm œ œ

7

‰ œ œ> œ œ .œ

#&

œétouffé avec papier sempre

p

œm

Ó

Ó ?

Œ ‰ jœb˝

‰ &6

˝superball

Vib.

p

3‰6˝ ‰

° sempre

ongles

%

%

6

® Tn -

pT JT Tb T

6

ym"f "

ym ym yy T- JT

%

Jœ

&

6 6 6 6 6>paume

ß

6

œ

pœ œ œm œb

œ œ

6æ j6æ

jyO>ß‰ 5& .

ym>

pplectre

y

˝

> . .y

yO

>

3

‰ œn ˝

® .

.œb˝

6&̋

6

œn ˝

& 6˝

.6˝

7

7

7‰ ® yæ gliss.(main)

arco

#IVyæ 3

yb y. y. y. y.%

6

Tym

ym y y y

6

yn ym ym yn ymT- JT Tb T .TbflM

ßTb Mflß

%

6

® œm œ

.œœ œb œ œ

œ .œ>7

œ .œ œ œ

7

‰ ® œ œb7

%

jyO

7

.yO

yb

#̋y

6

‰ ®6

6

6˝

œb ˝

6. ‰6

®œ˝

6>ß

28

1. snow melting on a metal roof 2. CataRT resynthesis with instrument sounds3. manually edited transcription 4. live reading by ensemble

51

New Concepts for Musical ApplicationsRe-arranging units by other rules than temporal order

Composition = Navigation through the sound space of heterogeneous sound databases

exploit richness of detail of recorded sound

efficient control by perceptually and musically meaningful descriptors

Interaction with self-recorded sound: sound of a musician is available for interaction beyond simple repetition

Cross-selection and interpolation: apply sound characteristics from one corpus to another, morphing between corpora

Orchestration and re-orchestration: matching a mass of sounds to a harmonic or sonic target while retaining precise control over the result

52


Exercise SuggestionsAdapting CataRT

Integration in your concert patchAdapting to your musical needs/ideas

Improving CataRTCorpus Data

soundfile editor (list/delete sounds in corpus)

soundset editor (define sound sets, add/remove units)

descriptor range viewer/editor (e.g.: exclude units below threshold)

Analysissegmentation editor

Selectionconstraints: don’t repeat unit

Synthesisforced pitch/loudness

multi-channel output

Controlby MIDI notes

by envelopes, sequencer

by live sound analysis

53


AcknowledgementsCataRT is based on FTM&Co by Norbert Schnell et al.Thanks go to:

Alexis Baskind, Julien Bloit, Greg Beller, Miguel Angel Ortiz PérezRoland Cahen, Yoan Olliver, Benjamin Wulf (ENSCI),Christian Jacquemin, Rami Ajaj (LIMSI)

Norbert Schnell, Riccardo Borghesi, Fréderic Bevilacqua, Rémy Muller, Jean-Philippe Lambert (IMTR Team)the French National Agency of Research ANR and the RIAM project Sample Orchestrator

Links:CataRT wiki on imtr.ircam.fr,

mailinglist [email protected] on http://lists.ircam.frand [email protected]

54

Documents

Advanced real-time sound and data processing with FTM&Coimtr.ircam.fr/imtr/images/Ftm_workshop_STEIM_2010_slides.pdf · 4. FTM & Co — IMTR Team — IRCAM - Centre Pompidou FTM Motivations