Marsyas

An open source platform for multimedia processing

http://marsyas.sourceforge.net

Nov 2010

Saturday, 20 November, 2010



Notice

This work is licensed under the Creative Commons Attribution-Share Alike 2.5 Portugal License. To view a copy of this license, visit

http://creativecommons.org/licenses/by-sa/2.5/pt/

or send a letter to

Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.




Marsyas Overview

Users & Applications

Usage Scenarios

Architecture

Interoperability

MarsyasX

Future Work

Summary





Marsyas OverviewSoftware framework for media analysis, synthesis and retrieval

Open source (GPL license)

Efficient and extensible framework design

original emphasis on Music Information Retrieval (MIR) ; now Multimedia!

C++, OOP

Multiplatform (Linux, MacOSX®, MS Windows®, …)

Provides a variety of building blocks for performing common audio tasks:

sound/video file IO, audio/video IO, signal processing and machine learning modules

blocks can be combined into data flow networks that can be modified and controlled dynamically while they process data in soft real-time.

WAV source

FFT

GMM

KNNLPC

WAV source

Hanning FFT

http://www.fsf.org


http://www.fsf.org

http://www.fsf.org

Marsyas OverviewMarsyas Brief History

1998 ~2000

Created by George Tzanetakis during his PhD activities at Princeton

2000 ~2002

Marsyas 0.1

First stable revisions of the toolkit

Distributions hosted at SourceForge

Creation of a developer community

User and Developer Mailing lists

2002 ~ …

Marsyas 0.2

Major framework revision

SourceForge SubVersion

http://marsyas.sourceforge.net/ http://www.cs.princeton.edu/~gtzan


http://sourceforge.net/projects/marsyas/

http://sourceforge.net/projects/marsyas/

http://www.cs.princeton.edu/~gtzan

http://www.cs.princeton.edu/~gtzan

Marsyas OverviewMarsyas Core Developers (Present and past...)

George Tzanetakis

Main culprit (also main designer and developer)

Luís Gustavo Martins

Refactoring freak

Graham Percieval

Documentation overlord

Luís Filipe Teixeira

Marsyas X motor-head

Mathieu Lagrange

Biggest network award

Steven Ness

Ruby and Web guy

Check updated list at the website!

http://marsyas.info/community/people




Marsyas Community

https://sourceforge.net/project/stats/graph/detail-graph.php?group_id=84982&ugn=marsyas&type=prdownload&mode=alltime&file_id=0&graph=1


http://sourceforge.net/project/stats/detail.php?group_id=84982&ugn=marsyas&type=prdownload




Marsyas Community

Google Analytics


Marsyas Community

Google Analytics(Nov 2010)


Marsyas CommunityMailing lists

Developer

https://lists.sourceforge.net/lists/listinfo/marsyas-developers

User

https://lists.sourceforge.net/lists/listinfo/marsyas-users

Tracker

Feature Requests




Marsyas Community

SourceForge Community Choice Awards 2009 Finalisthttp://community.sourceforge.net/cca_winners/bestacademia.html


http://community.sourceforge.net/cca_winners/bestacademia.html

http://community.sourceforge.net/cca_winners/bestacademia.html

Marsyas Project AnalysisMarsyas value?

Ohloh Analysis

http://www.ohloh.net/p/marsyas




Users & ApplicationsMusicream (Masataka Goto)

Music playback system with similarity capabilities

Uses Marsyas as its music similarity engine

http://staff.aist.go.jp/m.goto/Musicream/




Users & ApplicationsSndPeek (Princeton)

Uses Marsyas 0.1 for:

FFT magnitude spectrum

real-time spectral feature extraction

centroid

rms

flux

rolloff

http://www.cs.princeton.edu/sound/software/sndpeek/




Users & ApplicationsMarsyas @ INESC Porto

Audio Analysis Software prototypes:

Feature Extraction

Audio segmentation/classification

Audio fingerprinting

Speaker Segmentation

Music and Auditory Scene Analysis

Video Analysis Software prototypes:

Background modeling and subtraction

Local feature extraction for object matching

http://www.inescporto.pt/~lfpthttp://www.inescporto.pt/~lmartins

http://www.inescporto.pt


http://www.inescporto.pt/~lfpt

http://www.inescporto.pt/~lfpt

http://www.inescporto.pt/~lmartins

http://www.inescporto.pt/~lmartins

http://www.inescporto.pt/


Users & ApplicationsDesert Island

Undergraduate work at the Univ. Missouri

Kansas Jared Hoberock

Dan Kelly Ben Tietgen


Related WorkOpen Source frameworks

CLAM (http://clam.iua.upf.edu/)

STK (http://ccrma.stanford.edu/software/stk/)

Chuck (http://chuck.cs.princeton.edu/)

PureData (Pd) (http://crca.ucsd.edu/~msp/software.html)

Open Sound Control (OSC) (http://cnmat.berkeley.edu/OpenSoundControl/)

FAUST (http://faudiostream.sourceforge.net/)

EyesWeb (http://www.infomus.dist.unige.it/EywMain.html)

...

Commercial toolkits

MAX/MSP® (http://www.cycling74.com/)

MATLAB® Simulink® (http://www.mathworks.com/products/simulink/)

LabView® (http://www.ni.com/labview/)

DirectShow® GraphEdit (http://www.microsoft.com)

...


http://clam.iua.upf.edu/

http://clam.iua.upf.edu/

http://ccrma.stanford.edu/software/stk/

http://ccrma.stanford.edu/software/stk/

http://chuck.cs.princeton.edu/

http://chuck.cs.princeton.edu/

http://crca.ucsd.edu/~msp/software.html

http://crca.ucsd.edu/~msp/software.html

http://cnmat.berkeley.edu/OpenSoundControl/

http://cnmat.berkeley.edu/OpenSoundControl/

http://faudiostream.sourceforge.net/

http://faudiostream.sourceforge.net/

http://www.cycling74.com/

http://www.cycling74.com/

http://www.mathworks.com/products/simulink/

http://www.mathworks.com/products/simulink/

http://www.ni.com/labview/

http://www.ni.com/labview/

http://windowssdk.msdn.microsoft.com/en-us/library/ms787460.aspx

http://windowssdk.msdn.microsoft.com/en-us/library/ms787460.aspx

Usage ScenariosMarsyas command line tools

Demonstrate key capabilities of the framework

Some are actually research tools

Efficient and can execute in real-time

ANSI C++ only core

several optional libraries

MP3 reading/writing (libMad)

Tools and examples:sfplay

bextract

phasevocoder

sfplugin

…


Usage ScenariosPlaying audio files

sfplay

-s : start time for playback

-l : length of playback

-r : repeat times

-g : volume (gain) value

-p : playback.mpl (save playback network as a .mpl plugin file)

> sfplay foo.wav> sfplay –s 10.0 –l 3.2 –r 2.5 –g 0.5 foo1.wav foo2.au –f output.wav> sfplay –l 3.0 foo.wav> sfplay foo.wav –p playback.mpl>


Usage ScenariosMachine Learning: feature extraction and training of classifiers

bextract

-e STFTMFCC

extracts spectral and MFCC features

music.mf, speech.mf

lists of sound files (collections)

-w myWeka.arff

WEKA file with extracted features

-p ms.mpl

“trained” Marsyas plug-in for realtime music/speech classification

> bextract -e STFTMFCC music.mf speech.mf -p ms.mpl -w myweka.arff>


Usage ScenariosMarsyas plugins (.mpl files)

Allow to dynamically recreate a processing network in runtime

sfplugin

e.g. Audio playback

e.g. Realtime audio classification

> sfplugin –p ms.mpl unknownAudioSignal.wav>

> sfplugin –p playback.mpl foo.wav>


Usage ScenariosDigital Signal Processing

e.g. phasevocoder

e.g. onset detector

> phasevocoder –p 1.4 -s 100>

> mudbox -t onsets myMusic.wav>

http://eceserv0.ece.wisc.edu/~sethares/vocoders/phasevocoder.html




ArchitectureMarsyas 0.2

New dataflow model of audio computation

hierarchical messaging system used to control the dataflow network

inspired on Open Sound Control (OSC)

general matrices instead of 1-D arrays as data


ArchitectureMarSystem Slices

Separating things that happen at the same time from things that happen in different times

Correct semantics for

spectral processing


Implicit Patching VS Explicit Patching

Architecture

# IMPLICIT PATCHINGsource, F1, F2, F3, destinationFanout(F1, F2, F3)Series(source, Fanout, destination);

# EXPLICIT PATCHINGsource, F1, F2, F3, destination# connect the appropriate in/out portsconnect(source, F1);connect(source, F2);connect(source, F3);connect(F1, destination);connect(F2, destination);connect(F3, destination);


ArchitectureMarSystem Composites

Series

Fanout

Fanin

Parallel

Accumulator

…


Implicit VS Explicit Patching :: Neural Network

Architecture

#IMPLICIT PATCHINGfanoutLayer1(ANN_Node11, …, ANN_Node1N)…fanoutLayerM(ANN_NodeM1, …,ANN_NodeMN)ANN_Series(fanoutLayer1, …, fanoutLayerM)


ArchitectureTypical feature extraction

Time-frequency representation

Frequency summarization (MFCC)

Time summarization (Texture features)


ArchitectureFeature Extraction using Implicit Patching

MarSystemManager mng;

MarSystem* Series1 = mng.create("Series", “Series1");

MarSystem* Fanout1 = mng.create(“Fanout", “Fanout1");

MarSystem* Series2 = mng.create("Series", “Series2");

MarSystem* Fanout2 = mng.create(“Fanout”, “Fanout2”);

MarSystem* Fanout3 = mng.create(“Fanout”, “Fanout3”);

Fanout3->addMarSystem(mng.create(“Mean”, “Mean”));

Fanout3->addMarSystem(mng.create(“Variance”, “Variance”));

Fanout2->addMarSystem(mng.create(“Centroid”, “Centroid”));

Fanout2->addMarSystem(mng.create(“RollOff”, “Rolloff”));

Fanout2->addMarSystem(mng.create(“Flux”, “Flux”);

Series2->addMarSystem(mng.create(“Spectrum”, “Spectrum”);

Series2->addMarSystem(Fanout2);

Fanout1->addMarSystem(mng.create(“ZeroCrossings”, “ZeroCrossings”);

Fanout1->addMarSystem(Series2);

Series1->addMarSystem(mng.create("SoundFileSource",“Source"));


Series1->addMarSystem(mng.create(“Memory”, “TextureMemory”));


Series1->addMarSystem(mng.create(“classifier”, “Classifier”));


InteroperabilityMarsyas Audio and MIDI I/0

RtAudio

Multiplatform C++ API for realtime audio input/output

Linux (native ALSA, JACK, and OSS)

MacOSX®

Windows® (DirectSound® and ASIO®)

SGI®

RtMIDI

Multiplatform C++ API for realtime MIDI input/output

Linux (ALSA)

MacOSX®

Windows® (Multimedia Library)

SGI®

http://www.music.mcgill.ca/~gary/rtaudio/http://www.music.mcgill.ca/~gary/rtmidi/


http://www.music.mcgill.ca/~gary/rtaudio/

http://www.music.mcgill.ca/~gary/rtaudio/

http://www.music.mcgill.ca/~gary/rtmidi/

http://www.music.mcgill.ca/~gary/rtmidi/

InteroperabilityMarsyas & WEKA

WEKA: Data Mining Software in Java

Marsyas already includes some machine learning blocks

Marsyas outputs extracted features as .arff files (WEKA)

features can be opened in WEKA for further evaluation and data modeling

http://www.cs.waikato.ac.nz/ml/weka/




InteroperabilityCalling MATLAB® from C++ Marsyas code:

MATLAB® engine API

exchange data (i.e. matrices) in run-time between C++ and MATLAB®

remotely execute commands in MATLAB® from a C++ routine

Access to all MATLAB® toolboxes, algorithms and available routines

Algorithmic validation of C++ routines

Quick and easy evaluation of proof of concepts

May not allow real-time operation…

Not such a big problem when evaluating or developing algorithms

http://www.mathworks.com




InteroperabilityCalling MATLAB® from C++ Marsyas code:

Marsyas::MATLABengine class

Utility class

Wraps MATLAB® engine calls for most POD types and Marsyas data types

Easy to send/receive data to/from MATLAB® from anywhere in the code

// create a std::vector of real numbers std::vector<double> vector_real(4); vector_real[0] = 1.123456789; vector_real[1] = 2.123456789; vector_real[2] = 3.123456789; vector_real[3] = 4.123456789;

// send a std::vector<double> to MATLAB PUTVAR(vector_real, "vector_real");

// do some dummy math in MATLAB EVALUATE("mu = mean(vector_real);"); EVALUATE("sigma = std(vector_real);"); EVALUATE("vector_real = vector_real/max(vector_real);");

// get values from MATLAB double m, s; GETVAR(m, "mu"); GETVAR(s, "sigma"); GETVAR(vector_real, "vector_real");


InteroperabilityPython™ Bindings

easily create scripts for rapid testing and prototyping of data flow networks

would require much more development effort in C++

bonus: no compiling overheads

can also be embedded in C++ code, similarly to MATLAB® (TBD)

less tools for signal processing in general, but can be used for many other purposes (“batteries included”)

less licensing headaches

http://www.python.org




InteroperabilityMarsyas and Trolltech Qt4®

Qt® Core features optionally used by Marsyas

Multi-platform signal/slot architecture

Multi-platform threads (multithreaded processing)

Multi-platform database access

Multi-platform XML I/O

Qt® GUI Features optionally

used by Marsyas

Multi-platform Widgets

Multi-platform OpenGL

Qt4® is available as GPL open source code for all platforms

http://www.trolltech.com


http://www.python.org/

http://www.python.org/

MarsyasXWhat is MarsyasX?

Next step for Marsyas

Evolution rather than an alternative implementation

Generalization of the framework to support processing different media

Key points:

crossmodal processing

expandable support

interoperability

MPEGSink

GMM

MotionContour

WAV source

Hanning FFT AV

source

AV source

ContourMPEGSink


MarsyasXMarsyasX brief history:

MarsyasX started as a patch to Marsyas...

...add visual processing support

However, the previous architecture has some restrictions:

Some design options are audio-oriented

No in-place processing - for visual processing this can be too much of a burden

Representing complex data in a single buffer requires inefficient hacks

The slice-based processing paradigm is an elegant solution, but can be very difficult to adapt to a more generic framework


sink

MarsyasX - architectureFirst step was the creation of a generic data handling between modules:

Data carried in payloads

Flows define a coherent stream of data and are identified by type and name

Modules are explicitly associated to flows and can handle multiple flows with different behaviours (in, inout, out)

source sinkfiltersource filter


MarsyasX - architecturePayload Mechanism:

Payloads are created in factories, associated to a source;

Payloads are carried between modules through channels (created automatically by implicit patching);

When payloads are no longer necessary, they are sent back to its source to be reutilized -> memory is allocated only once.


MarsyasX - architectureControls are formally related to the flows and changes are propagated automatically

e.g. visual flow is characterized by width, height and colour space


MarsyasX - architectureTiming?

each “tick” corresponds to an elapsed time interval

synchronization is assured by the timing information associated to each payload

How is a module created?

define how many and what type of flows it supports

factories for outputs and flow-related controls are added implicitly

add specific controls, if needed

create a process function

input and output data structures are automatically passed as arguments


MarsyasX - architectureModules are now loaded from extensions

implemented separately

dynamically loaded

visual

core

application

audio learning avother

specific

marsyas


MarsyasX - interoperabilityPython

Scripting language plays a more central role

Complete bindings to create networks and use core functionalities

Marsyas 0.2

Legacy layer was created to support modules from Marsyas 0.2 transparently

A network can be an hybrid, containing MarsyasX and Marsyas 0.2 modules

MATLAB


MarsyasX - statusStatus of development:

pre-alpha

release 0.1 in mid-2011 (tentative)

feature status:

payload mechanism [done]

flow management [done]

definition of API [partly done]

python bindings [partly done]

legacy layer [partly done]

processing modules [needs a lot more]

event management (expressions, GUI) [not fully defined]


Main development trunk of the Marsyas project

Larger community:

associating the current users and developers of Marsyas with many others coming from different areas

Distributed MarsyasX

Data and events exchanged transparently

MarsyasX - future


Thank you!(Lifetime) Future work…





Technology

Marsyas