madBPM: A Modular Multimodal Environment for Data-Driven CompositionAnd Sonification

K. Michael Fox, Jeremy Stewart, Rob HamiltonRensselaer Polytechnic Institute

Department of the Arts110 Eighth Street

Troy, New York 12180foxk, stewaj5, hamilr4@rpi.edu

ABSTRACT

The modeling of biological data can be an inspirationalcompositional driver for artists seeking to create data-richmusical multi-media works. Working with the madBPM(Musical and Auditory Display for Biological PredictiveModeling) framework, composers and artists can quicklymap data from biological sources to structured sound andmusical process, as well as to visualization processes. ThemadBPM project couples real-time audio synthesis and vi-sual rendering with a highly flexible data-ingestion engine.Each component of the madBPM system is modular, allow-ing for customization of audio, visual and data-based pro-cessing. This paper details the inner workings of madBPM.

1. INTRODUCTION

Data sonification is a rapidly evolving discipline that ex-plores the use of data representing information from scien-tific processes, analytic models or real-time tracking as thedrivers for sound generating systems. The goal of manysonification systems is the conveyance and perception ofspecific information related to source data as realized us-ing sound as the principal component: an auditory display.In the same ways that computer-driven visualizations ofdatasets create visual analogues to the complex relation-ships between variables and data abstractions, sonificationsrepresent data as audible constructs existing in time. Forsituations involving real-time multi-modal monitoring ofcontinuous data streams, auditory displays working in con-junction with visual displays can provide additional atten-tional bandwidth for tasks ranging from the encoding ofavatar motion and action in virtual space [1, 2] to correct-ing athletes’ golf swings [3].

When sound is put to use as an expression of structureddata derived from a functional process, a significant trans-formational and translational component is necessary tomap parameters from this functional realm to parametersof sound that have the potential to express and expose spe-cific relationships of import within the data. As audiblesound itself is perceptually comprised of a great numberof parameters ranging from amplitude to frequency to tim-

Copyright: c©2017 K. Michael Fox et al. This is an open-access articledistributed under the terms of the Creative Commons Attribution License3.0 Unported, which permits unrestricted use, distribution, and reproduc-tion in any medium, provided the original author and source are credited.

bre to location in space, the organization of sounds andthe organization of the parameter mappings that are associ-ated with sound become crucial when attempting to createmeaningful sonifications of complex data sets, themselvesrepresentative of complex multivariate processes. Suchaesthetic and artistic exploration of data has great promisein driving new research paradigms as has been seen in re-cent projects begun as artistic initiatives such as StanfordUniversity’s “Brain Stethoscope” — a portable device thatdetects the onset of epileptic seizures through the real-timerhythmic and harmonic musical sonification of pre- andpost-ictal neural activity [4].

In this light, musical sonification can be understood asthe application of organizational techniques commonly at-tributed to musical composition and performance towardsthe grouping and structuring of sound generated from adata source. Musical characteristics such as pitch, rhythm,melody, harmony, timbre, tempo, spatialization, attack, de-cay and intensity can be mapped to raw parameters or cal-culated attributes of datasets, allowing for the creation ofsonic output that we hear as musical in nature while still ca-pable of conveying a great deal of information. Taken onestep further, musical structures, such as the transformationof organized or generated note material, or the applicationof continuous changes to instrumental parameters respon-sible for shaping the arc of a musical phrase can also bedriven by analyzed data.

Application of musical structures and techniques in thismapping process is experimental and compositional. Com-positional process focuses at different levels of detail at dif-ferent points in that process, ranging from low-level note-to-note attention to high-level structural attention. Simi-larly, sonification can range from mapping sound directlyon discrete data points parsed over time to sonification ofprocedures in transformational algorithms. Bovermann etal. [6] further distinguish between the sonification of al-gorithms by juxtaposing merely inputs and outputs versusinputs, outputs, and all intra-algorithmic transformationalsteps between. They also propose a distinction for opera-tor based sonification, where scientific models are directlyembedded in sonic mapping functions.

The importance of these distinctions becomes particu-larly clear when comparing sonification of processed, struc-tured data vs large sets of unstructured data. In the caseof unstructured data, there may not be an inherent modelwith which to scaffold sonic or visual mappings and anydata traversal method becomes an even more significant

Figure 1. An example of visualization in madBPM.

force in sonic signification. Significant questions aroundsoftware platforms for sonification and visualization in-clude whether the platforms are used for research-orientedexploration or post-research display of findings (presenta-tion). Another question is that of whether the end-user isseeking aesthetic exploration (artistic) or empirical knowl-edge (scientific) or some mix of the two. An ideal softwaretoolkit for data perceptualization would allow for produc-tive research and experiments by laboratories and artists,while also allowing that research to be presented live atprofessional meetings and artistic performances. The au-thors have produced a software toolkit and model for dataperceptualization that emphasizes user-defined “behavioralabstractions” to improve sonification software flexibilityand extensibility. This model has been implemented in themadBPM software platform to create a unified research en-vironment for both creative and analytical explorations ofdata through perceptualization.

2. MADBPM

The madBPM was initially designed as a data-ingestionframework for the multi-modal sonification and visualiza-tion of biological data, specifically data extracted from elec-trophoresis gels used in the synthesis of high-value chemi-cals like Heparosin. However, at the same time the projectwas intended to facilitate artistic expression of biologicaldata, i.e. musical sonification rather than straight auditorydisplay.

The madBPM software platform is built around a specificmodel that emphasizes actions and procedures. Sonficationand visualization in this platform is realized by end-userswho define code-based objects that describe data-flow andlogic in the traversal of data and mapping to sound or vi-

suals. The sound synthesis is provided by the SuperCol-lider sound and music programming language [5], but themodular design allows for different backends capable ofOpen Sound Control messaging to be used. The softwareis written in C++ and relies on openFrameworks 1 for thevisualization functionality.

We assume that musical structure and form can be uti-lized to not only represent characteristics of biological pro-cesses but more importantly also to aid researchers in dis-covering potentially interesting and important relationshipspreviously hidden within complex data sets. From this as-sumption, we designed madBPM, a modular sonificationand visualization platform that allows for the rapid pro-totyping and display of sonic and visual mappings. Ini-tially developed for a project focused on the identificationof key biological data points within the process of biosyn-thesis for high value chemicals, madBPM was designed asa modular toolkit, capable of interfacing with existing au-dio engines, visual coding languages and customized dataingestion modules. In its current state madBPM is linkedto the SuperCollider [SC] sound and music programminglanguage [5] and the openFrameworks visual programminglanguage 2 .

In the following sections, we describe some of the mostimportant architectural features of the madBPM softwareenvironment and data perceptualization model. These fea-tures are described in the context of the original researchproject from which the software environment emerged.

1 http://www.openframeworks.cc2 http://openframeworks.cc

Figure 2. madBPM GUI at startup. At left, the tagged data filteringpane. At center is the console output of SuperCollider. At the bottom isthe “utility bar”, featuring a button to build an active collection from theselected data tags, a Frames Per Second meter, and, at far right, a colorstatus icon indicating whether the platform has successfully initializedSuperCollider.

2.1 Data Perceptualization in madBPM

Auditory and visual mappings from datasets are experi-mentally and contextually derived. These mappings arefurther impacted by the initial state of the data being mapped—for instance, whether the dataset contains errors or invaliddata-points. Mappings may need to account for these, orthe data may need to be pre-filtered before sonification andvisualization. Software environments can position one orthe other approach as always necessary by design, preclud-ing the use of unstructured data sets. madBPM is designedto allow for flexibility in the kinds of data sets that mightbe processed in the environment by emphasizing its threelayers of behavioral abstractions: 1) program-level logic,2) data traversal/parsing, and 3) audio/visual mapping. Inthe madBPM software environment researchers, artists, orother users generate results by defining transformationalschemes at each of these three levels of abstraction. Inthe last layer, audio/visual mapping, data is transformedinto parameters of sonic and visual events. The secondlayer defines schemata for data-sets to be algorithmicallytraversed. In the final and “top” layer, changes in the twolower layers of abstraction can be automated. Each of theselayers is described more thoroughly in section 3.

2.2 Software Environment

madBPM makes extensive use of openFrameworks in itsarchitecture. Users of the platform are initially presentedwith a lean graphical user interface [GUI] comprised ofthree key components (Fig. 2). The first of the componentsis a pane displaying the structured data files automaticallyloaded at startup, described in Section 2.3. Each entry inthis pane represents a “tag” that describes a subsection ofthe data. Clicking on tags that appear in this list selectsand highlights that tag, while leaving tags that exist withinthat subset selectable and making tags not represented inthe subset unselectable (grey, non-clickable).

The second component, at center screen, is a GUI panelthat displays the console output of the managed SC pro-cess. This panel allows users to get reference informationfrom SC or debug unexpected behavior from within theplatform during any development or testing.

Figure 3. Detail of the tag list selection and filtering. Selecting tagsfilters out data that do not exist in all selected categories. Grey tags donot appear in the specified subset, while yellow tags may be selected tofurther constrain the set.

A third component of the platform interface features a“utility bar” like structure near the bottom of the screen.The GUI is extensible from source code and the utility baris a potential non-intrusive spot for buttons or short-handreference information. Currently, the bar features a colorindicator box representing the connection state with SC, aFrames Per Second meter, and a button “new collection”.Once a series of tags have been selected from the first GUIcomponent (the tag list), activating the “new collection”button triggers the data from those corresponding files tobe combined into a collection (ordered set) for sonificationand visualization. Visualization is drawn behind the GUI,which can be hidden by a hotkey.

2.3 Structure of Data

Data is currently read by the platform from partially pre-processed CSV files. These files provide both raw data inlabeled columns, but also tags that identify the relationshipbetween each file. From the research aims of the initialphases of the project, these files communicate analyzed in-formation from lanes on an electrophoresis gel image. Thelabels that identify the gel image and each lane within thatimage are the tags that are applied to files and subsets ofdata within those files.

When madBPM reads these files, they are organized in-ternally into “gel Lane” objects holding the raw data anddescribing their tag relationships to the platform. Aftertag filters have been applied and a new collection is con-structed, all internally stored gel Lanes matching the queryare aggregated into a “gel Collection” object that is usedby the platform for perceptualization. “gel Collection” ob-jects present themselves to the platform like a multidimen-sional iterator that are held by a Ranger object and parsedby Walker objects (Section 3). Once gel Lane objects arestored in memory, each gel Lane object merely wrap im-mutable data and are used to create user-defined gel Collectionobjects.

2.4 Sound Backend API

While the structure of visualization mappings are com-pletely internal to the C++ application core, sonificationmappings are represented within madBPM as objects intwo ways: Sounder objects in the platform’s C++ source

and corresponding sibling Sounder objects in SC source.Sounder objects are encapsulations of specific algorithmicbehaviors for the transformation of data into sonic or mu-sical material. To facilitate the communication betweenthe platform’s internal representation and the SC backendrepresentation, all data is transferred via Open Sound Con-trol UDP messages and conforms to a strict API. Messagesare directed to an address, which may be on the same com-puter or on another machine across a network, and are con-structed as a command string followed by correspondingarguments. All arguments after the command specifier aretagged by preceding the argument with a string descriptorbeginning with a ‘:’ character.

The application level commands sent from the core plat-form to SC are comprised of:

• loadSynthDefsLoad the platform’s SC synthesis definitions.

• create :cls :idCreate a new Sounder object of type :clswith class-specific unique :id.

• updateParams :cls :id :vol :rf :rv :mwUpdate the Sounder object of type :cls and :idwith the values :vol, :rf, :rv, and :mw.

• remove :cls :idRemove the Sounder object of class :cls and with:id.

• shutdownStop all active sounders and shutdown the process.

• funcDefinition :cls :selReply to madBPM front-end with a description of:cls’s function named :sel.

3. BEHAVIOR AND STRUCTURE IN MADBPMPROGRAMS

The most important design feature of the madBPM plat-form is that of behaviors. These are emphasized in threekey levels of abstraction: program-level logic, data-parsingbehavior, and data perceptualization algorithms or map-pings. Each of these levels of behavior are representedwithin the platform as objects which describe their func-tion over time (Fig. 5). From the beginning, madBPMwas intended to aid in lab research data-oriented artisticinquiry, but also to enable both real-time professional pre-sentation and artistic performance. Each of these levels ofbehavioral abstraction aim to address high and low levelstructural concerns for any of these contexts. Users de-fine the collections, subsets of the data based on selectedtags, to be sonified and visualized, and these collectionsare passed to the lower levels of the behavioral object hi-erarchy. By asking users to explicitly define the data pars-ing and meta structure (program-level logic), the platformis flexible enough to allow work with both un- or pre-processed datasets, structured or unstructured data, or mul-tiple forms of data segmentation and tagging. At the cur-rent state of the project, these objects are still defined inC++ source code and compiled into the platform.

3.1 Sounder Objects and Vis Objects

At the lowest level of the behavioral hierarchy are Sounderobjects and Vis objects. These objects define specific map-pings and algorithms for the transformation of data intovisual elements on the screen or sound through speakers.These objects do not traverse data, nor do they define therate at which data is accessed. These objects receive datafrom parent (Walker) objects and respond to them accord-ing to their defined behavior. All Sounder objects are poly-morphic relatives of a base Sounder class, while Vis ob-jects are similarly related to a VisObject base class. Bothclasses receive data from and interface with parents in iden-tical ways: Sounders and Vis objects respond to their par-ent Walker object’s call to an update function that acceptsall relevant perceptualization data. Since Sounder and Visobjects encapsulate self-contained visualization and soni-fication algorithms, these objects may range from simpleone-to-one mappings to much more complex real-time sta-tistical models.

3.2 Walker Objects

Walker objects are specifically encapsulated defined be-haviors for iteratively parsing gel Collections they refer-ence. Example behaviors might include: forward sequen-tial traversal, visiting each gel lane in sequence and everyvalue in the lane; reverse sequential traversal, opposite offorward sequential; minimum to maximum traversal, vis-iting elements across each or all lanes from lowest valueto highest; or, selective traversal, visiting every lane in thecollection and updates Sounders and Vis objects only forcertain values.

In the hierarchy of behavioral objects, Walkers commu-nicate the values they visit in the data with the Sounder andVis objects to which they have references.

3.3 Ranger Objects

Ranger objects encapsulate “program-like logical structure”,and interface directly with the platform and Walker objectsthey own. Ranger objects are analogous to the role of amusical ensemble conductor. Typically, only one Rangerclass would be active at any given time, and these classesdefine meta-level structures and sequences during a profes-sional or artistic presentation of the data perceptualization.A Ranger object, for example, might begin its operationby defining three different concurrent Walker objects andafter some conditions have been met replace two of themwith Walker objects of different behavior types.

These objects create the Walker objects that traverse thedata, and choose which perceptualization objects those Walkerobjects should report to — creating, removing, or alteringthe relationships between these when necessary.

4. AN EXAMPLE PROGRAM

This section will describe an example hierarchy of behav-ioral abstractions that would define a specific operatingprogram in the madBPM platform. As described in theprevious section, Ranger objects encapsulate the storageand lifespan of objects that parse data streams, or the pro-gram logic of the presentation. For this example program,the Ranger object might begin operation by automatically

Figure 4. A closeup of the the team’s “Norris” Vis object. Data from electrophoresis gels are used to transform and extrude 3D meshes in spaces,representing density and skew as signifiers of specific trials and the distribution of their molecular weights.

Figure 5. The behavioral object hierarchy. At the top level, the plat-form references a Ranger object, which defines a “program-like logic”executing over time. Ranger objects own at least one Walker object, eachparsing data collections. Walkers communicate the data they parse to theSounder objects and Vis objects they own.

creating two collections of data from different subsets oftags. The collections (A and B) might consist of all of thegel Lanes tagged with “supernatant”, “M9pG”, ”hrs6”,while the second consists of gel Lanes tagged with “Cel9AT”,“lipA”, “Induced”, respectively. Next, the Ranger mustassociate each collection with a parsing object. CollectionA could be assigned a ForwardWalker and collection B aLocalMaxWalker.

The ForwardWalker and LocalMaxWalker objects are pre-defined built-in Walker objects for traversing data collec-tions they are associated with. ForwardWalker’s parse theeach of the gel Lanes in their associated collection in theorder they are defined, and within each gel Lane this walkervisits each datum in the order defined. LocalMaxWalkervisits each gel Lane in its associate collection in the or-der defined in the collection similar to the ForwardWalker.However, for each gel Lane visited, the LocalMaxWalkerwill only visit the largest local data value. The rate atwhich Walkers traverse the collection they are associated

with is also specified by the Ranger object that definedthem. The parent Ranger also specifies the action thatWalker objects should take when they have reached the endof their collections. By default, Walkers that reach the endof their collection return to the start and continue parsingagain. But Walkers can also be set to stop all parsing uponcompletion and to tell their parent Ranger that they havefinished.

Instead of discontinuing parsing, the example Ranger willallow the default looping behavior and keep track of itsown timing clock. Now that the program consists of sub-sets of data and behavioral abstractions that define how toparse them, the Ranger must associate sonification and vi-sualization mappings for the Walkers. Perceptualizationalgorithms are encapsulated in Sounder and VisObjects.The example Ranger will create a ScaleSounder for bothWalker objects, but it could provide ForwardWalker with aGelBars VisObject (Fig. 1) and a LocalMaxWalker with aNorrisMesh VisObject (Fig. 4). The drawn output of bothvisualization objects are overlaid on the same screen space.Based on each Walker object’s specified sample timing, theobjects poll their next data point and deliver that data totheir connected mapping objects, both Sounders and Vi-sObjects.

In this example, the Ranger might use an internal timer tosequence changes in data collections, parsing algorithms,and perceptualization mappings. For example, after 5 min-utes have elapsed the Ranger could fade out the Scale-Sounder attached to the LocalMaxWalker and replace itwith a TimbreShapeSounder. After a few more moments,the example Ranger might remove ForwardWalker fromthe program, replacing it with a slightly altered clone ofthe LocalMaxWalker.

It is also possible for the program flow in the Rangerto change based on conditional logic. An example men-tioned above involved possibly removing a Walker once it

had completely parsed a data collection instead of loopingagain over the data. Another possibility, however, is thatWalkers that encounter data values within specific rangescould instigate structural changes in their parent Ranger.For example, if a Walker iterating over a collection encoun-ters a value that is near a given molecular weight and hasa localized intensity above a given threshold, the Rangercould respond by removing some Walkers and generatingother new collections, parsing algorithms, and data map-pings. The resulting change affects both the aural and vi-sual data mappings, but also the logical structure of theanalysis program. This flexibility could possibly providea means of exploring perceptual feature optimization auto-matically through behavioral objects.

5. CONCLUSIONS AND FUTURE WORK

The artistic nature of musical sonification is a key elementin the future plans for the madBPM platform, allowingour team to pursue both artistic and diagnostic goals us-ing the project. Working in conjunction with members ofthe Biological Sciences department at RPI our team is in-vestigating methods of multi-modal sonification and visu-alization to allow researchers to better understand relation-ships between proteins used in the synthesis of high-valuechemicals. madBPM allows both researchers and artiststo process and map data parameters from recent experi-ments quickly and efficiently to parameters of sound rang-ing from low-level synthesis techniques to higher-level or-ganizational or compositional parameters. In this mannerwe envision a series of sonification and visualization ex-periments that analyze data sets from multiple viewpoints.

At the same time, the use of biological data as the pro-genitor of data-driven artworks is central to the madBPMproject, allowing composers and visual artists to experi-ment with biological data in the creation of multi-modalartworks. An exhibition of such works is currently be-ing planned at RPI’s Collaborative-Research AugmentedImmersive Virtual Environment [CRAIVE] Laboratory toshowcase how biological data can inspire art, as well ashow art can inspire research using biological data. WithinmadBPM’s technical implementation, the project has clearlydefined future milestones including:

• Implementation of a Domain Specific Language (DSL)for real-time scripting and definition of Rangers, Walk-ers, and Sounder/VisObjects

• Support for real-time data streams and ad-hoc datamodels

• Expanding the existing support for running the plat-form as a networked application

The creation of a DSL for real-time scripting of the madBPMplatform will allow for more rapid research prototypingand expressive aesthetic data explorations. Currently, theteam has proposed a DSL model that would focus on defin-ing the high level behavioral abstractions, allowing usersto customize data mapping and program logic at runtime.The scripting language specification would require a bal-ance between low level access to data and functions thatcan reach the level of composition — creating complex be-haviors from underlying behavioral components.

madBPM uses a data model that is derived from elec-trophoresis gels in biosynthetic chemical research. In thecourse of implementing a data importing framework, theteam explored many useful ways of tagging, storing, andpassing data around the platform. Another proposal for fu-ture development is the generalization of methods for dif-ferent types of data models and streams. In particular, theteam would like to implement a pipeline for real-time datastreams with data models that can be defined or redefinedon the fly (in coordination with the DSL milestone).

The final proposed milestone for future development isexpansion of madBPM’s networking capabilities. As thesize of data archives and complexity of algorithms grow, itoften becomes necessary to distribute computational work-loads amongst networked computer nodes. This is espe-cially true for professional and aesthetic presentation ofdata perceptualization, where timing and reliability can becrucial. madBPM’s use of Open Sound Control alreadyleverages a balance in speed and reliability and the pos-sibility of communicating across a network between frontand back ends. Proposed future development would breakprocesses down into smaller units for concurrent distributedcomputation, allowing visual and audio display to be bro-ken up between several monitors and speakers.

6. ACKNOWLEDGEMENTS

The work described here was made possible through a grantfrom the Rensselaer Polytechnic Institute’s Knowledge andInnovation Program (KIP).

7. REFERENCES

[1] R. Hamilton, “Musical Sonification of Avatar Physi-ologies, Virtual Flight and Gesture,” Lecture Notes inComputer Science: Computer Music MultidisciplinaryResearch (CMMR) Journal, Springer-Verlag, Heidel-berg, Germany, 2014.

[2] R. Hamilton, Perceptually Coherent MappingSchemata for Virtual Space and Musical Method,Ph.D. Thesis, Stanford University, 2014.

[3] M. Kleiman-Weiner and J. Berger. 2006. “The SoundOf One Arm Swinging: A Model For Multidimen-sional Auditory Display Of Physical Motion.” Interna-tional Conference on Auditory Display, London, UK.http://www.mit.edu/ maxkw/pdf/kleiman2006sound.pdf

[4] G. Slack, “Hearing a Seizure’s Song”, Discover Mag-azine, online: http://discovermagazine.com/seizure,May, 2014.

[5] T. Bovermann, J. Rohrhuber, and A. de Campo, “Lab-oratory Methods for Experimental Sonification,” inT. Hermann, A. Hunt, and J.G. Neuhoff, editors,The Sonification Handbook, Logos Publishing House,Berlin, Germany, pp. 237–272.

[6] J. McCartney, “Rethinking the Computer Music Lan-guage: SuperCollider”. Computer Music Journal, vol.26, no. 4, pp. 61–68, Feb. 2002.