A TEMPO- AND TRANSPOSITION-INVARIANT PIANO MUSICCOMPANION

Andreas Arzt1, Cynthia C. S. Liem2, Gerhard Widmer1,31Department of Computational Perception, Johannes Kepler University, Linz, Austria

2Delft University of Technology, Delft, The Netherlands3Austrian Research Institute for Artificial Intelligence (OFAI), Vienna, Austria

andreas.arzt@jku.at

1. INTRODUCTION

In [1] a prototype of a piano music companion was pre-sented that is able to follow and understand (at least tosome extent) a live piano performance. Within a few sec-onds the system can identify the piece that is being played,and the position within the piece. It then tracks the progressof the performer over time via a robust score following al-gorithm and continuously re-evaluates its current positionhypotheses within a database of scores. Thus, it is capa-ble of detecting arbitrary ‘jumps’ by the performer (e.g.,leaving out repetitions, re-starts at any position) – not onlywithin a piece, but within the complete database of clas-sical piano scores. The companion can be used in multi-ple ways, e.g., for piece identification, music visualisation,and automatic page turning, both during piano rehearsaland during live concerts.

For this demonstration, we extended this prototype withthe ability to cope with arbitrary transpositions by the mu-sician. This is achieved by using a transposition-invariantfingerprinting approach presented in [1], in combinationwith the automatic adaption of the underlying score fol-lowing algorithm to possible transpositions.

2. SYSTEM OVERVIEW

Figure 1 gives an overview of the piano music compan-ion. The system is provided with a database of sheet mu-sic in symbolic form. The companion is listening to theperformance via a single microphone. An on-line pianomusic transcription algorithm based on a recurrent neuralnetwork [3] is used to translate the audio stream into sym-bolic information (a list of pitches with timestamps).

The extracted notes of the last few seconds of the liveperformance are then matched to the database of sheet mu-sic via a tempo- and transposition-invariant fingerprintingmethod [1]. This process is continuously running in thebackground, regularly providing new piece and (rough) po-

c© Andreas Arzt1, Cynthia C. S. Liem2, GerhardWidmer1,3.Licensed under a Creative Commons Attribution 4.0 International Li-cense (CC BY 4.0). Attribution: Andreas Arzt1, Cynthia C. S. Liem2,Gerhard Widmer1,3. “A Tempo- and Transposition-invariant Piano Mu-sic Companion”, 15th International Society for Music Information Re-trieval Conference, 2014.

Possible Applications

Live Performance

'Any-time' On-line Music TrackerInstant Piece Recognizer

Piano Music Transcription Algorithm(On-line Audio-to-Pitch

Transcriptor)

Tempo- and Transposition-

invariant Symbolic Music Matcher(Fingerprinter)

Multi Agent Music Tracking System

Output: Score Position

Musical Score Database

Figure 1. System Overview

sition hypotheses for the multi-agent tracking component.In addition, these hypotheses also contain information on apossible transposition, i.e., how many semitones lower orhigher than notated in the score the piece is being played.

The multi-agent tracker consists of multiple instancesof a score following algorithm based on audio-to-audiomatching via DTW [2]. Using the provided hypotheses,each instance is matching the current live audio stream to aposition in the database of sheet music, taking into accountpossible transpostitons. At any point in time, the positionof the highest scoring agent (i.e., the one resulting in thelowest alignment costs) is reported as the current positionin the database.

This prototype enables multiple applications. A basicuse case is music identification, where, given only the liveaudio stream, the piece that is being played is identifiedand the metadata (e.g., composer and title) is returned. Thesystem can also be used as a basis for music visualisa-tions and performance enrichments, ranging from a simplemarker showing the score position, to more sophisticatedenrichments, like showing information about the structureof the piece and the most important themes, and givinghints about what to listen for at specific moments. Further-more, the companion can be used during piano rehearsaland during piano concerts by the performer for showingthe sheet music, without the need to turn the pages manu-ally.

3. DEMONSTRATION

The demonstration is designed as a brief live show witha professionally trained pianist at the piano. In this set-ting we will demonstrate the capabilities of the companion(music tracking, music identification, transposition- andtempo invariance) in a coherent way. After some preparedexamples, to explicitly demonstrate these capabilities, wewill invite the audience to participate in the demo and se-lect the pieces for the pianist to play. The duration of thisshow is about 15 to 20 minutes. Afterwards, we would liketo encourage members of the audience to try our piano mu-sic companion. We are very interested in their feedback.

For the live demonstration we ask the organisers to pro-vide us with:

• a piano (either an acoustic piano or an electronic oneplus loudspeakers)

• a projector

• a microphone

• a stage or room, where we can play the piano withoutdisturbing other demos

We will bring all the remaining equipment.

4. ACKNOWLEDGMENTS

This research is supported by the Austrian Science Fund(FWF) under project number Z159 and the EU FP7 ProjectPHENICX (grant no. 601166).

5. REFERENCES

[1] A. Arzt, S. Bock, S. Flossmann, H. Frostel, M. Gasser,and G. Widmer. The complete classical music compan-ion v0. 9. In Proceedings of the 53rd AES Conferenceon Semantic Audio, 2014.

[2] A. Arzt and G. Widmer. Simple tempo models for real-time music tracking. In Proceedings of the Sound andMusic Computing Conference (SMC), 2010.

[3] S. Bock and M. Schedl. Polyphonic piano note tran-scription with recurrent neural networks. In Proceed-ings of the International Conference on Acoustics,Speech, and Signal Processing (ICASSP), 2012.