Matteazzi Master's Thesis 2009

DIPARTIMENTO DI INGEGNERIA DELL’INFORMAZIONE

CORSO DI LAUREA SPECIALISTICA IN INGEGNERIA INFORMATICA

TESI DI LAUREA

MUSIC PERFORMANCE WITH DELAYED AUDITORY

FEEDBACK: AN EMBODIED PERSPECTIVE

RELATORE: Prof. Giovanni De Poli

CORRELATORE: Prof. Marc Leman (Ghent University)

LAUREANDO: Marco Matteazzi

Padova, 20 ottobre 2009

ANNO ACCADEMICO 2008 – 2009

2

© Copyright

by

Marco Matteazzi

2009

3

dedicata a mia madre, a mio padre, e ad Elisa

4

5

Noi non abbiamo un corpo, noi siamo un corpo.

We don’t have a body, we are a body.

Pier Paolo Pasolini

6

7

ABSTRACT

Research on music playing with delayed auditory feedback (DAF) shows that timing

asynchronies between action and perception profoundly impair performance, to the

extent that musical execution may be interrupted. New technologies for music pro-

duction are often affected by significant latencies, causing playing music to be diffi-

cult and unsatisfactory. This scenario calls for the search of useful techniques for

playing music in presence of DAF, overcoming the difficulties introduced by the

mismatch between action and perception. In this thesis, I approach this issue from an

ecological, embodied perspective, considering the role of human body as a mediator

between the sonic energy in the environment and the inner world of the performer. I

investigate the effects of DAF on piano performance under three different auditory

conditions (normal auditory feedback, absence of auditory feedback, DAF), focusing

on variations in MIDI parameters and in performers’ body movement, captured

through sensors. Results confirm that DAF significantly impairs music performance

while absence of auditory feedback does not. Moreover, a diminution in body

movement is found in the absence of feedback, whereas under DAF embodied re-

sponses seem to depend strongly on the players’ personal attitude.

8

9

ACKNOLEDGMENTS

First, I’d like to thank my supervisor prof. Giovanni De Poli for giving me the op-

portunity to undertake this extraordinary experience.

I am very grateful to all the people at IPEM. Primarily, I whish to thank prof. Marc

Leman for his great helpfulness and for the continuous stimuli his ideas provide to

me. I especially thank the three people who helped me more during this work, Mi-

chiel Demey, Frank Desmet, and Dirk Steenbrugge. The first for his indispensable

support; the second for the great job he did with the ANOVA analysis of Paragraph

5.1, as well as for having introduced me to the world of statistics (“there are three

kinds of lies: lies, damned lies, and statistics”); the third for his precious assistance

with regard to pipe organs. I whish to thank the other people at IPEM who kindly

helped me in some occasions: Ivan Schepers, Micheline Lesaffre, Pieter Coussement

and Frederik Styns.

I’d like to thank all the students of the Systematic Musicology class 2008 at Ghent

University for their hospitality and their sympathie. In particular, I’d like to thank the

three special friends with which this project started: Imke De Hert, Renske Wit-

teveen, and Pieter-Jan Maes.

I thank very much all the people who helped in some ways during this work: Jan

Vanlerberghe, Ivo Delaere, Bart Meynckens, Tine Allegaert, Lukas Huisman, Clara

Van der Bremt, Frederic Lamsens, Marianne Van Boxelaere, Anke Steenbeke,

Herman Streulens, Yves Senden, Delphine Grandsart, Kersten Cottyn, Johan

Wijnants, Lize Raes, Giovanni Bruno Vicario, Massimo Grassi, Federico Avanzini,

Enrico Marchetto, Luca Mion.

Finally, I’d like to thank Valentina Munaro who beautifully corrected the thesis, Ni-

cola Barban and Elisabetta De Cao for their improvised statistics assistance at the No

Dal Molin Festival (and not only), Lucie Jurystova for the picture in Figure 1.1, and

Alessio Guzzano for making me know the Pasolini’s quote.

10

11

CONTENTS

Abstract 7

Acknoledgements 9

Chapter 1 - Introduction 15

1.1 Music performance

1.1.1 Music performance and sensory feedback

1.1.2 Music performance as a timed sequence of motor acts

14

14

18

1.2 The role of feedback in movement execution

1.2.1 Closed-loop vs open-loop models

18

18

1.3 Auditory feedback in music performance

1.3.1 Feedback deprivation

1.3.2 Delayed auditory feedback (DAF)

Critical interval vs relative time hypothesis

Break-point interval

Theoretical implications

1.3.3 The role of auditory feedback in musical sequence production

19

20

21

22

23

24

25

1.4 New scenarios for the research on music performance affected by

DAF

26

1.5 An ecological, embodied approach to the study of music perform-

ance with DAF

27

1.6 Thesis organization 30

Chapter 2 – Empirical review of music performance with DAF 31

2.1 Literature survey 31

2.2 Summary

2.2.1 Accuracy

2.2.2 Timing

2.2.3 Dynamics

2.2.4 Expressivity

35

36

36

37

37

12

2.2.5 Body movement 37

Chapter 3 - Method 39

3.1 Subjects 40

3.2 Stimulus material 41

3.3 Conditions 44

3.4 Equipment 44

3.5 Procedure 46

Chapter 4 – Data analysis 49

4.1 MIDI data 49

4.2 Movement data

4.2.1 Synchronization of MIDI and movement data streams

4.2.2 Intensity of movement

4.2.3 Head pitch periodicity

51

51

52

53

Chapter 5 – Results 55

5.1 One-way ANOVA of tempo, dynamics, and intensity of movement

against the experimental condition as factor

5.1.1 Exploratory data analysis

5.1.2 Homogeneity of variances

5.1.3 ANOVA table

5.1.4 Tamhane's T2 post hoc test

5.1.5 Homogeneous subset tables

5.1.6 Means plots

5.1.7 Summary of the one-way ANOVA results

55

55

58

58

61

64

65

67

5.2 Correlations 67

5.3 Timing and dynamics profiles 70

5.4 Individual results

5.4.1 Tempo and velocity


5.4.3 Periodicity of the head pitch movement

72

72

76

77

13

Chapter 6 – Discussion and conclusions 81

6.1 Timing 81

6.2 Dynamics 82

6.3 Expressivity 83

6.4 Body movement 83

6.5 Conclusions 85

Chapter 7 – Reference bibliography 87

Appendix 1 – Individual subjects’ graphics 97

A1.1 Subject 2 98

A1.2 Subject 3 100

A1.3 Subject 4 102

A1.4 Subject 5 104

A1.5 Subject 6 106

A1.6 Subject 7 108

A1.7 Subject 8 110

A1.8 Subject 9 112

A1.9 Subject 10 114

A1.10 Subject 11 116






Appendix 2 – Measurement of the delay times of the church organ of

St. Anna in Ghent

129

A2.1 Method 132

A2.2 Results 135

A2.3 Discussion 138

A2.4 Conclusions 139

A2.5 Reference bibliography 140

14

15

CHAPTER 1 - INTRODUCTION

Music performance is a highly skilled human activity in which action and perception

are tightly coupled. While playing music, if the primary brain area dedicated to mo-

tor control activates, the auditory area activates too, and vice versa. Coupling of ac-

tion and perception permits the musician to feel immersed in the environment, and to

be able of modifying it to his or her taste just moving on the instrument. Such experi-

ence of immersion strongly relies on sensory feedback, which contributes to the per-

ception of the action-relevant values of the physical energies in the environment. In

particular, auditory feedback has an important role in the matching of produced ac-

tions and perceived ones. In fact, as many studies on delayed auditory feedback

(DAF) report, alterations of auditory feedback timing that introduce asynchronies be-

tween action and perception can profoundly impair performance, to the extent that

musical execution may be interrupted: this phenomenon seems due to the need for

congruency between what is produced and what is perceived.

New technologies for music production are often affected by significant DAF, caus-

ing the musical performance to become more difficult and unsatisfactory. Examples

may be suites for network playing, in which the large communication distances pro-

duce unavoidable latencies. This scenario calls for the search of useful techniques for

playing music with DAF, overcoming the difficulties introduced by the mismatch be-

tween action and perception. I will approach this problem from an ecological, em-

bodied perspective, considering the role of human body as a mediator between the

sonic energy in the environment and the inner world of the performer, made of inten-

tions, meanings, and significations.

In this chapter, I introduce music performance as one of the ultimate human motor

skills (Paragraph 1.1), requiring high motor and cognitive capabilities, and strongly

relying on sensory feedback (Subparagraph 1.1.1). Subparagraph 1.1.2 briefly pre-

sents the cognitive view of music performance as a timed sequence of motor acts:

according to this approach, research on feedback results central in understanding

perception-action coupling mechanisms. In Paragraph 1.2, I discuss the role of feed-

back in the execution of motor acts in general: open-loop and closed-loop models of

motor control are presented. In Paragraph 1.3, I focus on the role of auditory feed-

16

back in music performance, reporting the results coming from investigations on mu-

sic performance without auditory feedback (Subparagraph 1.3.1) and under DAF

(Subparagraph 1.3.2). Subparagraph 1.3.3 resumes these findings. Paragraph 1.4

shows how latency affects many electronic instruments and interfaces, so that DAF

contexts are frequently found in the field of music production. Paragraph 1.5 presents

the adopted ecological, embodied approach to the study of DAF affected music per-

formance. Lastly, Paragraph 1.6 describes the organization of the next chapters.

1.1 Music performance

Music performance is a highly skilled activity that involves both cognitive and motor

capabilities and demands for a strict connection between them. The ability of skilled

musicians to coordinate fine body movements to produce complex meaningful se-

quences is often considered as one of the ultimate examples of human motor skills

(Bernstein, 1967; Lashley, 1951). Some musicians are capable of carrying into action

extreme tasks, showing impressive abilities in the control of hand and finger move-

ments. Skilled pianists, for example, can produce movements at rates that exceed

visual reaction times (e.g., in the execution of trills; Lashley, 1951), playing up to 30

sequential notes per second for sustained periods (Rumelhart & Norman, 1982).

High-level playing is based on long-term and intensive rehearsal of motor patterns,

that has the aim of forming an inner space of automatic motor trajectories to be re-

called and generated without paying too much conscious attention to them (Leman,

2007). Decades of regular practise are necessary to completely automate the motor

patterns: the hours of training needed can be roughly estimated at 10,000 (Ericsson,

Krampe, & Tesch-Römer, 1993; Howe, Davisdon, & Sloboda, 1998). The acquired

experience permits good players to focus on musicality - the transmission of the ex-

pressive intentions - rather than on movements and technique. Thoughts, emotions,

aesthetic forms and ideas can thus be communicated to the audience through sounds.

1.1.1 Music performance and sensory feedback

In music performance, a strict correspondence between player’s intentions and gen-

erated sound is necessary for the transmission of the correct musical message. There-

fore, musicians continuously monitor their performance through sensory feedback,

17

also when they’re playing solo. Some musical activities like group playing (e.g., or-

chestral music) require in addition the player to coordinate with other musicians, so

that visual and auditory feedback plays a more central role (for the interdependency

between musicians during ensemble performances see Rasch, 1979; 1988). In some

playing styles based on group improvisation (e.g., jazz ensembles, see Figure 1.1)

feedback is absolutely necessary: musicians are often called to decide in real-time

what and how to play basing themselves on the information they collect about what

their colleagues are playing at the same time (e.g., in standard jazz improvisation,

“interdependent routines such as call and response, propagating motifs, supporting

and contrasting dialogs, and a higher level of leader/follower dynamics”; Weinberg,

2002, p. 21). These observations point out the necessity of investigations of the role

of feedback in music performance: is feedback always necessary for playing? Is

feedback used for error correction? Are musicians able to cope with distorted sensory

feedback? In the following paragraphs, relying on the results in literature, I will try to

answer these questions limiting to the case of solo music performance, in which

feedback is not used for inter-subjects coordination.

Figure 1.1 A jazz quartet performance. In improvised group music auditory and vis-

ual feedback have a very central role.

18

1.1.2 Music performance as a timed sequence of motor acts

A common approach to the study of human serial behaviours like music performance

or speaking implies a focus on the role of feedback. This kind of serial behaviours

can be viewed as examples of timed sequences of motor acts: research on feedback

investigates how such timed sequences are actually produced, which is indicated by

Lashley (1951) as one of the central problems of cognition. The manner in which

perceived consequences of actions influence the production of subsequent ones is

studied, with the aim of clarifying both planning and production mechanisms. In

general, understanding the role of feedback is crucial for the comprehension of the

relationships between action and perception (see Pfordresher, 2006): this two com-

ponents of human behaviour are simultaneously involved in sequence production,

and require a certain degree of congruency to keep the production fluent and correct.

In the last analysis, understanding the role of feedback is important to clarify the

complex relationships between the inner subjective world of human beings, ex-

pressed through action, and the external reality (even if, as we will see, such a strict

subdivision between inner and outer is somewhat outdated).

1.2 The role of feedback in movement execution

1.2.1 Closed-loop vs open-loop models

The control of motor acts such as musicians’ movements can be explained by two

main theoretical models: the closed-loop model, in which, during a movement, feed-

back is used to control if the goal is being achieved, and the open-loop model, in

which feedback is not used for correctness control. In the closed-loop model, sensory

feedback is necessary since movement control is totally depending on the peripheral

information (feedback control hypothesis): the execution and the completion of an

action is guided by a centralized comparison between the intended movement and the

feedback information. Vice versa, in the open-loop model, execution is centrally

leaded by an abstract representation of the motor sequence stored in memory (motor

program), and feedback can have a role in determining or triggering possible re-

sponses, but not in the guidance of the current movement. An early variant of the

open-loop theory is the response-chaining hypothesis (James, 1890), in which

19

movement is composed by a chain of muscular contractions, and the feedback from

one contraction (response-produced feedback) serves as a stimulus for the next con-

traction in the chain. In this approach, sensory feedback, as a trigger, is necessary for

the execution of the movement, in contrast with other open-loop theories according

to which feedback doesn’t have a role at all in the execution phase. It may be of in-

terest for the present discussion to notice that the response-chaining hypothesis pro-

vides an account for the timing among the contractions, very important in skilled ac-

tivities like music performance: such timing, called relative timing, would be deter-

mined by the temporal delays in the various sensory processes (i.e., responses propa-

gation).

Each motor control theory relies on some experimental evidences and is opposed by

others. In general, closed-loop theories give the better account for longer duration

movements, such as driving a car, in which the possibility of error correction is evi-

dent, whereas open-loop theories seem more suitable for explaining rapid move-

ments. The hypothesis of the cohabitation of different types of motor control relies

on the limited velocity of feedback transmission. In fact, though some studies

(Bowman & Combs, 1969; Cohen, Goto, Shanzer & Weiss, 1965; Fuchs & Korn-

huber, 1968) have shown that some kinds of very fast responses – 5 to 10 ms – are

possible, the elapsed time between the error detection and the start of the correction

is estimated on average at 200 ms. During the execution of fast movements, a re-

sponse time of 200 ms is too long to permit feedback to have an active role in error

correction, as stated by closed-loop theories: open-loop theories seem therefore

likely, at least for this class of movements.

1.3 Auditory feedback in music performance

In order to investigate the role of feedback in sound production tasks like music and

speech, audition is the most studied feedback channel. In these kind of tasks, in fact,

the output of the system is sound: therefore, though in music performance visual, tac-

tile and proprioceptive feedback are very important (for the importance of vision in

music performance see Sloboda, 1982; Banton, 1995), audition is the only feedback

channel which consents a direct comparison between the produced action (e.g., a

keypress) and the desired goal. Experimenting with auditory feedback is therefore an

20

appropriate solution to test whether open-loop or closed-loop systems are involved in

music production.

The role of auditory feedback in sound sequence production has been investigated

mainly through two experimental approaches. The first one implies a study of the ef-

fects of feedback removal: an impairment of production, in the absence of feedback,

would indicate the necessity of auditory feedback, at least for a correct realization of

the production aspects disrupted by such absence. A second approach consists in

studying the effects of feedback alteration: each sound resulting from a produced ac-

tion is altered, so that the coordination of the auditory feedback with actions is modi-

fied. This experimental paradigm is known as altered auditory feedback (AAF). Al-

terations can occur in the dimensions of time and pitch, where timing alterations in-

troduce an asynchrony between the onset of a produced action and the onset of the

corresponding feedback, whereas in pitch alterations the pitch associated with a pro-

duced action is not the pitch that is normally associated with that action. Both kinds

of alteration can be simultaneously present, giving rise to a combination of onset

asynchrony and unexpected feedback content. The AAF paradigm means to investi-

gate the relationships between action and perception, and, in particular, to answer the

questions about how action and perception are bounded together.

1.3.1 Feedback deprivation

Studies on feedback deprivation seem to show that auditory feedback is not strictly

necessary in music sequence production. In fact, though auditory feedback is shown

to be important in the learning phase (Finney & Palmer, 2003; Highben & Palmer,

2004), other researches indicate that its absence doesn’t significantly impair the pro-

duction of learned sequences (Gates & Bradshaw, 1974; Banton, 1995; Finney, 1997;

Repp, 1999), even for untrained performers (Pfordresher, 2005). However, auditory

feedback may still be necessary in some kinds of fine control, since Repp (1999) re-

ported small effects of its absence on expressive parameters of production. The fact

that auditory feedback doesn’t appear to be necessary in music performance supports

Lashley’s open-loop theory (Lashley, 1951), that, based on the trilling speed of con-

cert pianists, argued for the impossibility of a role of feedback in motor control dur-

ing very fast movements (see also Keele, 1968). However, in the above mentioned

studies on auditory feedback deprivation, the remaining feedback channels were not

21

inhibited, so that it could be argued that visual, tactile and proprioceptive feedback

can still guide players’ movement for a correct execution. Nevertheless, feedback

does not have much time to affect execution, which testifies against this hypothesis,

as well as the fact that some kind of activities seem to be executable in absence of

kinesthetic feedback (Keele & Summers, 1976; Lashley, 1951).

To sum up, relying on the results in literature, it seems likely that motor acts in music

performance are memorized through sensing during training, to form an inner space

of motor trajectories (Leman, 2007). These trajectories can be recalled without the

aid of auditory feedback, even if, in this case, a small degradation of fine perform-

ance parameters is possible. For what concerns motor control, a motor program

(open-loop) model is adopted, at least for fast movements: comparison between

feedback and intended results may be used to adapt subsequent actions, but not to

guide the current. On the other hand, closed-loop models can be applied to slower

movements, for which error correction is possible.

1.3.2 Delayed auditory feedback (DAF)

The most extensively studied AAF paradigm consists of introducing a certain delay

between the onset of a produced action and the onset of the corresponding feedback:

this experimental condition is called delayed auditory feedback (DAF). It is well in-

vestigated how DAF strongly disrupts sequence production in many tasks including

music performance, speech, and rhythmic tapping. Two early studies on DAF speech

(Lee, 1950; Black, 1951) reported significant slowing of production rate, increased

sound level, and increased articulatory errors, with a predominance of insertions and

repetitions. Many studies confirmed these and other negative effects of DAF on the

various kind of sequence production tasks. A review of the studies on music per-

formance under DAF is given in Chapter 2.

The introduction of a delay between note onsets and feedback onsets may lead to

three different situations, depending on the relationship between the amount of delay

(delay length) and the inter-onset-intervals (IOIs) duration. First, when the delay

length is shorter than the IOIs duration, the feedback onset of a produced event i oc-

curs before the produced event 1i : in this case, only the timing of production is

altered. The second case occurs when the delay length is equal to the IOIs duration,

so that the feedback onset of a produced event i is simultaneous to the produced

22

event 1i : in such situation, only the pitch contents are altered. The third case oc-

curs when the delay length is longer than the IOIs duration, and the feedback onset of

a produced event i succeeds the produced event 1i : in this case, both timing and

pitch alterations are present.

In general, DAF disrupts both temporal relationships between note onsets (timing)

and notes correctness (accuracy). Recently, Pfordresher (2003) showed that altera-

tions of feedback timing disrupt the timing of sequencing more than the accuracy,

whereas alterations of feedback content without asynchrony (e.g., pitch alterations

that occur with DAF when the delay time is equal to the IOIs duration) disrupt accu-

racy but do not influence much timing. These findings suggest that a strict connec-

tion exists between the disrupted aspects of the performance (i.e., timing or accuracy)

and the kind of feedback alteration (i.e., timing or pitch alterations).

Critical interval vs relative time hypothesis

A frequently discussed topic regarding DAF disruption is the kind of dependency

from the amount of delay, and, in particular, the amount of delay that causes maxi-

mal disruption. In the early studies, disruption caused by DAF was found to increase

with the amount of delay up to a certain point, called critical (delay) interval, or de-

lay of maximal impairment, and then to reach asymptote (e.g., in music, Gates, Brad-

shaw, & Nettleton, 1974, found an asymptote around 270 ms,) or to decrease (e.g., in

speech, Fairbanks & Guttman, 1958). These findings have given credit to the so

called absolute time hypothesis, according to which the delay of maximal impairment

occurs when the absolute temporal separation between a produced action ant its

feedback onset falls within a certain temporal window, regardless of the production

rate. Anyway, all experiments on music performance supporting this view have been

using fixed delay lengths, so that, in case of tempo variations or of non-isochronous

pieces, the phase relationships between the timing of produced actions and their rela-

tive feedback onsets would not be constant. Moreover, production rate was usually

not controlled. Recently, Pfordresher & Benitez (2007), using a kind of delay called

adjustable delay, which roughly preserve the phase relationships between actions and

feedback onsets, found that disruption is best predicted by the relative phase location

rather than by the absolute position of feedback onsets. This result gives support to

23

the relative time hypothesis, which states that perception and action are coordinated

according to the rhythmic cycles formed by IOIs (cf. Jones, 1976; Robinson, 1972).

Break-point interval

Another discussed issue concerning DAF is which is the minimum delay length

which causes auditory feedback to be actually perceived as delayed and DAF disrup-

tion to become significant. This topic is related to the investigation of temporal

thresholds in perception and cognition. In a classical study on vision, Card, Moran &

Newell (1983) showed that, if two events are connected by an immediate causality

relationship and the perception of the second is progressively delayed, degradation of

immediate causality starts for some subjects as early as 50 ms. Moreover, they found

that, while the perception of immediate causality ends around 100 ms, perception of

delayed causality begins at 50 ms, reaches a peak around 100 ms, and terminates

around 160 ms, threshold after which the events are recognized as independent. Re-

search on the temporal ordering of two distinct stimuli showed that the events re-

quired a minimum of 30 ms to be perceived as successive, regardless of sensory mo-

dality (Poppel, 1997). In the field of music perception, Rasch (1979) observed that

listeners often judge ensemble performance as synchronous despite asynchronies of

30-50 ms.

For what concerns playing an instrument under DAF, Finney (1997) reported that

professional pianists may perceive delay lengths of under 10ms, so that this threshold

is often suggested as the maximum latency for a music controller (Finney, 1997;

Freed, Chaudhary, & Davila, 1997). However, a certain degree of tolerance to higher

latencies is well-documented. Dahl & Bresin (2001), in a study on synchronization

under DAF, individuated between 40 and 55 ms a possible break-point at which DAF

begins to make the performance increasingly difficult. Mäki-Patola & Hämäläinen

(2004), testing the threshold of latency tolerance of subjects playing a theremin (a

continuous sound instrument without tactile feedback), reported that latency started

to be perceived at 30 ms, when comparing to a reference with zero latency. With this

delay length, subjects perceived latency with a high degree of uncertainty; conscious

detection was found to start at 60 ms.

Other studies addressed the break-point interval problem for network duet perform-

ances. In this situation, differently from solo playing, inter-subjects coordination is

24

required, so that the effect of DAF is supposed to be even stronger. However, results

do not differ from what found as for solo playing. Chew, Sawchuk & colleagues

(Sawchuk et al., 2003; Chew et al., 2004) found latency tolerance in network playing

to be dependent on both the piece and the instrument played. In general, in such

situations, they suggested a general threshold for latency tolerance at 50 ms. Chafe,

Gurevich, Leslie, & Tyan (2004), quantifying the effects of latency on rhythmic

clapping network performance, showed that the performance is at its best when the

round-trip bi-directional latency is comprised between 20 and 30 ms, and degrades

with higher latencies.

Basing on these empirical results, the interval of delay lengths in the range of 30-60

ms seems therefore to be a plausible break-point for the correct execution of music

performance under DAF. A certain variability is to be taken into account, mostly due

to the kind of instrument and the piece played.

Theoretical implications

In a recent review, Pfordersher (2006) resumed the empirical results on music per-

formance under AAF, and DAF in particular, discussing their theoretical implica-

tions. What clearly emerges from studies on DAF is that, despite the fact that audi-

tory feedback is not necessary for musical sequence production, a certain match be-

tween auditory feedback and produced actions is required when feedback is present.

In other words, congruency between action and perception is needed: miscoordina-

tion between them causes disruption of performance. In particular, it seems that high

impairment from DAF is due to the fact that the feedback sequence is equal in struc-

ture to the planned events sequence, but it is also in conflict with it, since each feed-

back onset occurs simultaneously to the production of events with a different posi-

tion in the planned sequence. In other words, it is likely that DAF disrupts production

by virtue of the interfering effect of perception of events planned for the past on the

current activation of subsequent events for production. This view is strengthened by

research on AAF paradigms different from DAF: manipulations of feedback contents

resulting in a sequence of events highly dissimilar to the planned sequence (extrane-

ous feedback) do not disrupt performance as much as DAF does (Gates & Bradshaw,

1974; Finney, 1997; see Finney 1999 for a review of similar results in speech pro-

duction). Further support comes from the reduced disruption reported by Finney

25

(1997) and Pfordresher (2003) for combined alterations of timing and pitch: instead

of a summation of the different effects of asynchrony and changed contents, overall

impairment decreased, suggesting that combined alterations cause the feedback se-

quence to be perceived as unrelated to the planned sequence.

Given these findings, Pfordresher (2006) hypothesizes a new theoretical framework

in which action and perception share a common representation of sequence structure

in memory guiding both the planning of actions and the interpreting of the perceived

consequences of those actions. This theory is consistent with neurophysiological evi-

dence for the so-called mirror neurons, that respond in the same way when humans

or monkeys execute an action and when they perceive the same action executed by

another individual (Rizzolatti, Fogassi & Gallese, 2001; see Leman, 2007, pp. 95-96,

for a brief summary on this topic). Moreover, Pfordresher’s model assumes a basic

functional separation between timing and sequencing (see also Krampe, Mayr &

Kliegl, 2005; MacKay, 1987), thus accounting for the fact that different kinds of

feedback alteration cause different kinds of disruption.

Similarly to the research on feedback deprivation, studies on AAF provide evidence

against the feedback control theories. In fact, if feedback was used for error correc-

tion, any kind of feedback alteration would signal that an error has occurred, thus

causing disruption of performance. Instead, as said, not all alterations of pitch con-

tents strongly affect production. In Pfordresher’s opinion (Pfordresher, 2006), the

problem with classical closed-loop theories is that they focused on the relationships

between feedback and planning limiting the field to individual events, whereas it is

plausible that these relationships involve sequences of actions and concurrent feed-

back sequences.

1.3.3 The role of auditory feedback in musical sequence production

To conclude, research on music performance shows that, in general, auditory feed-

back does not function as a “feedback”, since it is not strictly necessary for musical

sequence production; rather, it is a “recurrent auditory information” (Howell, 2004)

that facilitates learning (Finney & Palmer, 2003) and control of fine nuances of per-

formance (Repp, 1999). However, studies on DAF show that a congruence between

auditory feedback and actions is needed. Citing Pfordresher (2006, p. 195), this con-

gruence perhaps reflects “a more general sensitivity to statistical regularities in the

26

environment, including the relationships between actions and correlated perceptual

events”. In fact, according to motor program theories, an internal model of move-

ment trajectories guides the execution of actions, relying on the expected conse-

quences of these actions: incongruences between perceived and planned conse-

quences of actions disrupt production, presumably because of the shared representa-

tion of action and perception in memory. In conclusion, “the way in which people

coordinate perception and action during music performance builds on a general ten-

dency for people to relate action plans to perceptual information” (Pfordresher, 2006,

p. 195).

1.4 New scenarios for the research on music performance affected by

DAF

Normally, as for traditional acoustic instruments, feedback onsets are perceived as

simultaneous with the actions generating them: the sound is directly produced by the

player’s movements, without time-consuming technological mediations, and it is

generated close to the player, at distances on the order of few decimetres or few me-

tres. One remarkable exception is given by church organ, in which mechanical fac-

tors may cause considerable delays – sometimes of hundreds ms (see Appendix 2) -

between the player’s actions and the diffusion of sound. An example of source of la-

tency in the mechanics of church organs is the pressure transmission from the key-

board to the pipes, especially in the organs in which such transmission is pneumatic,

rather than mechanical or electronic.

Recent developments in digital technologies have opened new prospects in which a

DAF condition may be inevitable. In the last two decades, a proliferation of music

instruments and interfaces in which the auditory content is processed, or transmitted

over a network, has forced musicians, instrument makers and developers to cope

with significant latencies: as with church organs, amount of DAF may be on the or-

der of hundreds ms (e.g., in the field of vocal controllers, Hämäläinen, Mäki-Patola,

Pulkki, & Airas, 2004). An important and promising context in which latency is nec-

essarily present is that regarding systems for collaborative network playing such as

duet at a distance (Sawchuk et al., 2003; Chew et al., 2004; Chafe, Gurevich, Leslie,

& Tyan, 2004; Bartlette et al., 2006; Kapur, Wang, Davidson, & Cook, 2005; for re-

27

views see Barbosa, 2003; Follmer, 2002; Weinberg, 2002). Other examples of la-

tency-affected systems for music production may be physical models instruments,

gestures-driven systems for sound generation and control, handheld devices wire-

lessly linked to the Internet and audio services (see Smith, 2001), vocal controllers

for edutainment (Hämäläinen, Mäki-Patola, Pulkki, & Airas, 2004) or sound synthe-

sis (Smith, 2001; Janeer, 2008). In such systems, the introduction of delays is mainly

due to two kinds of constraint (cf. Smith, 2001): first, physical constraints limit the

speed of signal transmission through the various media; second, practical limits af-

fect the speed of signal processing. In suites for network playing, transmission con-

straints are often predominant over processing limits, due to the long distances and to

the fact that round-trip latency also has to be taken into account (Kapur, Wang,

Davidson, & Cook, 2005). Instead, in other latency-affected systems like those cited

above, computation time is usually predominant (e.g., see Smith, 2001). The total la-

tency of a system is given by the sum of all the delays from the source to the user.

Sources of latency are numerous: examples are delays due to conversions (e.g., ana-

logue to digital), features extraction (e.g., pitch detection from an audio stream),

buffering (e.g., of input samples in audio drivers and APIs), internal latencies of sub-

parts of the system (e.g., of applications or sensors), operating systems activities

(e.g., context switching, inter-thread communication). The total latency of a system

may be variable in time, especially in the case of systems for network playing, for the

reason that “long-distance networks with Internet Protocol (IP) routing often result in

asymmetry, jitter and packet loss” (Chafe, Gurevich, Leslie, & Tyan, 2004, p. 4).

1.5 An ecological, embodied approach to the study of music per-

formance with DAF

We saw in Paragraphs 1.3.2 and 1.3.3 that studies on DAF give support to the hy-

pothesis according to which perception and action share a common representation in

memory. This hypothesis merges into the more general notion of coupling of percep-

tion and action, on the basis of which perception it is not merely the sensing of the

physical properties of reality (distal stimuli) through the effect of these properties on

sensory input (proximal stimuli; Brunswik, 1956); instead, perception is considered

as a simulated action, because aspects of the outer world are directly captured in

28

terms of their action-relevant value (called affordance; Gibson, 1979). Duality be-

tween an individual and its environment is therefore overcome by the fact that the

individual can have a direct access (i.e., not mediated by inference and judgement) to

the energies in the environment. This view relies on the observation according to

which knowledge of the outer world itself does not emerge from passive perception,

“but from the need to act in an environment” (Leman, 2007, p.43; this point is well

illustrated by a classic study on kittens by Held & Hein, 1963). The ability to relate

sensory features with the cause that generated them is therefore due to the fact that

perception and action in individuals are not only linked, but they evolved together:

“cognitive structures emerge from the kinds of recurrent sensorimotor pattern that

enable action to be perceptually guided” (Varela, Thompson & Rosch, 1991, p.176).

Many experimental results are giving support to the idea that perception and action

are inseparable in lived cognition, to the extent that they share a common neuronal

events code (cf. Hommel, Müsseler, Aschersleben, & Prinz, 2001). Research areas

go from observations on imitation in newborn infants to the discovery of the mirror-

neurons in human and monkeys (for a brief review of results in these fields, see

Leman, 2007, pp. 89-91). In particular, a tight coupling of perception and action is

observed in brain activity during music performance: when the primary motor area is

activated, the primary auditory area activates too, and vice versa (Lotze et al., 2003;

Kristeva et al., 2003; Langheim, Chakarov, Schulte-Mönting, & Spreer, 2002;

Hickok, Buchsbaum, Humphries, & Muftuler, 2003; Haslinger et al., 2005). Playing

music seems therefore “embedded in a goal-directed ontology of involvement with

music” (Leman, 2007, p. 96). Research on DAF, as we showed, brings further argu-

ments in favour of these views.

Given that cognition of human beings is determined by the need to act in an envi-

ronment, with bodies gifted with sensorimotor capabilities and subjected to social

and cultural constraints, Varela, Thompson & Rosch (1991) suggested that the study

of cognition should concern not the recovery or projection of physical features, but

the embodied action. This theoretical approach, called embodied cognition paradigm,

overcomes the classic cognitive tradition, “criticized for its neglect of the action

component in the subject’s involvement with the environment” (Leman, 2007, p. 43).

For what concerns the study of music, Leman (2007) transferred the embodied ap-

proach of Varela, Maturana, and others, to the systematic musicology methods, giv-

29

ing born to the paradigm called embodied music cognition. In embodied music cogni-

tion, individuals engage with music thanks to the fact that, while perceiving sonic

energy, they simulate or imitate moving sonic forms and corporeal articulations:

these evoked bodily gestures have a meaning for the listener “due to his or her per-

sonal history as an active participant within a cultural environment” (Keller & Janata,

2009, p. 289). The aspects of the outer world, music included, are captured in terms

of embodied resynthesis. Thus embodied music cognition focused on the relation-

ships between subjects and their cultural and physical environment, and can be there-

fore considered an ecological approach to music research.

As the term “embodied” underlines, human body has a central role in this paradigm:

physical body is considered as a mediator between the mind, which sees music in

terms of intentions, meanings, and significations, and the energies in the environ-

ment, sonic and not. The possibility of a direct perception of the affordances in the

environment through body gestures causes musical involvement to be based on cor-

poreal articulations. Thereby, the study of all musical activities is centred on the role

of body movement. Among the aims of the embodied approach to musicology is the

search for a mediation technology that links bidirectionally musical energies to ges-

tures. The search is motivated by new digital technologies, in particular real-time in-

teractive music systems (e.g., sensor-based instruments) and information retrieval

techniques.

In light of what is said above, I will approach the study of music performance under

DAF from an embodied perspective: namely, I will investigate the role of body

movement in DAF affected music performance, in particular in helping the musician

to keep the desired timing profile. The hypothesis is that under DAF musician will

attempt to accentuate the feedback modalities important for performance which are

not altered, i.e. visual, tactile and kinesthetic. Stressing movements, concentrating on

the notation, increasing playing intensity may be useful strategies to give less impor-

tance to the auditory channel, through which the misleading information is received.

I will check for possible changes in the periodicity of body movement, in order to in-

vestigate the effect of DAF on this aspect of corporeal articulation. Moreover, I will

verify whether there are individuals capable of playing fluently under DAF, and, in

case, which strategies they adopt. For this reason, some of the participants in the ex-

periment were chosen among expert church organists, who are used to play with

30

relevant amount of DAF. The underlying motivation for this research is that embod-

ied strategies may be an important aid to all the musicians who encountered signifi-

cant DAF while performing on latency-affected electronic interfaces and instru-

ments, such as those mentioned in Paragraph 1.4.

1.6 Thesis organization

The organization of this thesis is the following: Chapter 2 summarizes the existing

empirical results on music performance under DAF. Chapter 3 describes the experi-

mental method followed. In Chapter 4, the extraction of the parameters to analyze is

explained. Chapter 5 provides the experimental results, while Chapter 6 discusses

them and draws some conclusions. Chapter 7 reports the reference bibliography. Two

appendices conclude the thesis: Appendix 1 reports graphics concerning the experi-

mental results of the single subjects, while Appendix 2 describes the measurements

of the amount of DAF of St. Anna church organ, in the city of Gent, Belgium.

31

CHAPTER 2 – EMPIRICAL REVIEW OF MUSIC

PERFORMANCE WITH DAF

In this Chapter, I summarize the experimental results concerning music performance

with DAF. Paragraph 2.1 provides a survey of the existing studies, briefly indicating

the performance parameters studied, the stimulus materials, the experimental condi-

tions, the relevant points about the procedures, and the salient results. In the survey,

only the studies in which the stimulus is a musical piece or a melody are considered:

single-note or tapping performances are not taken into account. In Paragraph 2.2 a

summary of the results is given, considering each of the following aspects of music

performance: accuracy, timing, dynamics, expressivity, body movement.

Before proceeding, I will provide some clarifications about the glossary adopted in

the following sections and in the rest of the thesis. With normal condition, I refer to

an experimental condition in which the auditory feedback is not altered, in contrast

with altered auditory feedback paradigms such as DAF or extraneous auditory feed-

back (cf. Paragraph 1.3). Analogously, silent condition refers to an experimental

condition of feedback deprivation, whereas delayed condition to a DAF paradigm.

Lastly, key velocity (or simply velocity) indicates the MIDI parameter that represents

the force with which a keyboard key is pressed, thus providing a measure of the play-

ing dynamics.

2.1 Literature survey

The first studies on the effect of DAF on music performance were carried out by

Kalmus, Denes & Fry (1955) in a research on clapping under DAF. In their work,

they reported that “exploratory experiments showed that whistling and the playing of

musical instruments were, in fact, strongly influenced by delayed acoustic feed-

back”. The first research explicitly designed to investigate this issue was accom-

plished by Havlicek (1968), who tested the effect of DAF on the number of errors

during sight-reading of unfamiliar musical compositions, as well as the difference in

susceptibility among woodwind, strings, brass and piano performers. With a delay

time of approximately 200 ms, performance under DAF was found to be louder than

32

in a normal condition, it showed increased IOIs, and included more errors. Perform-

ance was disrupted for all instruments used.

Gates, Bradshaw and colleagues have performed a number of studies concerning

DAF and music playing. As reported by Finney (1999) in his review, in their ex-

periments subjects were asked to play “as fast as possible” a practiced piece from no-

tation. The dependent variable was the total elapsed time necessary to perform the

piece. In a study focused on the left-right differences between ears, Bradshaw, Net-

tleton & Geffen (1971) tested the effects of different delays on a piano performance

combining normal auditory feedback with DAF. 200 ms delay was found to be more

disruptive than 400, 550, 750 and 1100 ms delays. Gates & Bradshaw (1974), study-

ing subjects performing an etude on an electronic organ, found that a performance

under 180 ms DAF took longer than a performance with normal feedback. Another

aim of this study was to find out whether a difference between genders subsist in

auditory perception, with the result that no significant difference was found. Gates,

Bradshaw and Nettleton (1974) compared the effect on keyboard performance of 12

different delay times, equally spaced between 100 and 1000 ms, in a task combining

normal auditory feedback with DAF. They found that disruption in total elapsed time

increased with delay length and reached asymptote at 270 ms. In this regard, it is im-

portant to point out that the individual rate of playing was not controlled, though a

post-hoc analysis suggested that the slower and the faster players were similarly af-

fected. Another result was the reported tendency to repeat individual notes or to in-

sert an extra note that, for instance, extended a scale passage one note beyond its cor-

rect conclusion. Insertions and repetitions never comprised more than a single note.

This fact can be correlated with the increased total elapsed time, insofar as insertions

and repetitions can be viewed as two of the factors which cause the performance to

be completed in a longer period of time.

Finney (1997) examined performances of Bach pieces by trained pianists under 250

ms DAF condition. Individual subjects were asked to play at their preferred tempo

“without expressive variations”. Performances were significantly disrupted by DAF

in different ways including note errors, total elapsed time, key velocity and interhand

coordination: more errors were made, production rate was slower, and key velocity

was higher. Approximately 60% of the total note errors under the DAF condition

were insertions. Finney also studied the case of DAF mixed with random pitch al-

33

terations, finding a reduced impairment in comparison with the simple DAF condi-

tion.

Pfordresher and colleagues have performed a number of studies on the coordination

of perception and action in music performance, providing different auditory feed-

backs to subjects playing on an electronic keyboard. In contrast with the methods of

the previous research, which used real piano pieces as stimulus material, they chose

simple isochronous melodies of 8 or 12 quarter notes to be performed with the right

hand only, appositely “designed to be easy to produce and repeatable without

changes in hand position”. The musicians were asked to play the melodies in a “flat”

way (mechanically), so that possible increased timing variability could be considered

as an evidence of disruption.

Pfordresher & Palmer (2002) focused on the effect of DAF on the timing of music

performance, and in particular on the phase relationships between produced onsets

and auditory feedback. In Experiment 1, they had pianists performing melodies at

two production rates (600 ms and 500 ms IoIs) with different amounts of DAF (0,

100, 150, 200, 250, 300 and 350 ms). Timing variability of performances was meas-

ured through the coefficient of variation, CV (standard deviation of IOIs/mean IOI).

The results show a general increase in timing variability with the amount of delay,

with a relatively reduced timing variability for .5 phase ratio (twice the amount of

DAF) for the 500 ms IOIs production rate (but not for the 600 ms). In Experiment 2,

pianists had to choose the preferred tempo with 200, 250, 300 and 350 ms delays.

Performers chose slower rates for larger delays, with the preferred rate approximated

twice the amount of DAF (.5 phase ratio). In Experiment 3, Pfordresher & Palmer

tested whether instructions to mentally subdivide produced events in two or three dif-

ferent blocks lead to a reduced disruption. The experiment included four delay condi-

tions (none, 200, 300 and 400 ms) at the 600 ms IOIs production rate. As in Experi-

ment 1, they found that temporal variability increased with DAF. A second relevant

finding was that a deliberate subdividing reduced the timing variability with longer

feedback delays. Finally, they failed to demonstrate a reduction of DAF disruption

when feedback onsets coincided with planned subdivisions, in contrast with the re-

sult of reduced disruption for .5 phase found in Experiment 1.

Pfordresher (2003) used a new kind of delay, named phase shift (or adjustable de-

lay), that adjusted to produced timing so that feedback onsets maintained a roughly

34

consistent relative phase position within produced IOIs. In Experiment 1, phase

shifted tones occurred after their associated keystrokes but before the subsequent

ones. The phase shifts used were of .33, .50 and .66, with respect to the expected

IOIs. Results showed that, in comparison with a normal feedback performance, phase

shifts increased timing variability and slowed production rate, while error rates were

only marginally increased. However, error rates did increase with the amount of

phase shift, with most errors occurring with a .66 phase shift. In Experiment 2, audi-

tory feedback onsets and produced keystrokes were synchronous, but feedback

pitches were altered to match pitches associated with earlier keystrokes by a lag of

one, two or three events (serial shift). Alterations of feedback pitch without asyn-

chrony were found to increase errors but not to influence produced timing. Experi-

ment 3 incorporated combined phase and serial shifts, which caused a moderate dis-

ruption of timing and accuracy and revealed interactive effects of serial and phase

shifts on production.

Pfordresher & Palmer (2006) explored the effects of serial shifts on music perform-

ance in case of pitches matching events intended both for the past (delays) and for

the future (prelays). The trials tested the feedback distance of 1 to 3 events in both

feedback directions (past and future). All alterations disrupted the accuracy of the

performance more than timing. Overall disruption was not influenced by feedback

distance or by whether feedback events originated from past or future events. The

specific kind of errors, anticipatory or perseveratory, was found to depend on feed-

back direction: future feedback increased perseveratory errors, while past feedback

increased anticipatory errors.

Pfordresher & Benitez (2007) investigated whether disruption from DAF reflects the

phase location of feedback onset or the absolute temporal separation between actions

and sounds. Non skilled participants had to play simple isochronous melodies or to

tap an isochronous beat at the production rates of 330, 500 and 660 ms IOIs. In Ex-

periment 1 fixed delays of 330, 500 and 660 ms and adjustable delays of 66%, 100%

and 132% of the produced IOIs were used. Experiment 2 presented the same condi-

tions as Experiment 1, except for shorter delays, appositely “designed to form a dis-

tribution around lengths that should cause maximal disruption according to the abso-

lute time hypothesis” (see Paragraph 1.3.2): 165, 250 and 330 ms for fixed delays

and 33%, 50% and 66% of produced IOIs for adjustable delays. For both delay types,

35

results showed that disruption was best predicted by the phase location of feedback

onsets, and it decreased when feedback onsets formed harmonic phase ratios (phase

synchrony). In contrast with a finding of Pfordresher & Palmer (2002, Experiment

2), no relative advantage was found for .5 phase ratio. Finally, different movement

tasks (melody production versus tapping) led to slightly different patterns of disrup-

tion across phase.

Moelants, Demey and Leman (2009), in a research for which the present study was a

preliminary investigation, had 10 professional musicians performing four different

pieces of piano music on an digital piano, with DAF of 200 and 300 ms. The pieces

were “Für Elise” by L. van Beethoven, “Sonatina op.36/1” by M. Clementi, “Bulgar-

ian Rhythm (Microcosmos 113)” by B. Bartok, and “Sarabande in g-minor” by G. F.

Händel. The analyzed aspects were dynamics, tempo per measure, amount of asyn-

chrony (between notes notated on the same point in the score), errors (order errors,

deletions, wrong notes, insertions), and upper body movements. In all pieces they

reported a significant increase, with the DAF conditions, in both key velocity and

measures duration. In three pieces out of four, other significant differences with re-

spect to the normal condition were observed: increased measures duration variability,

increased amount of asynchrony, with a larger asynchrony in the 300 ms delay, and

increased errors (insertions in particular). In the slowest piece, the Händel Sarabande,

no significant effects of DAF on these parameters were found. Head movements

showed a significant increase in the delayed condition in three of the pieces played,

with exception of the piece by Clementi.

2.2 Summary

All the studies evidenced that DAF profoundly disrupts performance, “to the extent

that a skilled performer sounds like a beginner” (Pfordresher 2006). I will now sum-

marize the reported effects of DAF on some specific aspects of music performance -

keyboard performance in particular - in comparison to normal auditory feedback per-

formances. These aspects include accuracy, timing, dynamics, and performer’s body

movement.

36

2.2.1 Accuracy

Many studies in which fixed delays and real musical pieces were used (Havlicek,

1968; Gates, Bradshaw & Nettleton, 1974; Finney, 1997; Moelants, Demey &

Leman, 2009) reported an increased number of pitch errors in performance under

DAF, with a preponderance of insertions/repetitions. Other kinds of pitch errors ac-

centuated by DAF are deletions, wrong notes, order errors (Moelants, Demey &

Leman, 2009). It seems therefore proved that DAF disrupts the accuracy of music

performance. Despite that, Pfordresher (2003, Experiment 1) showed that the number

of errors in an performance increased only marginally with .33, .50 and .66 phase

shifts. A likely explanation for this apparent contradiction is that Pfordresher used

adjustable delays instead of fixed ones, as well as simple isochronous melodies in-

stead of real pieces, reaching thus a rough control over the phase location of the

feedback onset, otherwise difficult to guarantee. It seems indeed that error rates in-

crease with phase shift from the expected feedback onset (Pfordresher 2003, Experi-

ment 1), and that significantly higher error rates are found when feedback onsets are

synchronous with subsequent produced events (serial shifts, see Pfordresher, 2003).

Therefore, the position of feedback onsets relative to produced time intervals influ-

ences the number of errors more than their absolute temporal position (cfr.

Pfordresher & Benitez, 2007; the dependence of error rates on the piece-delay length

combination in Moelants, Demey, & Leman, 2009, seems to give further support to

this thesis).

2.2.2 Timing

All previous studies reported that DAF disrupts timing aspects of music perform-

ance. In particular, a striking and well-documented effect of the DAF condition on

instrument playing is the slowing of the production rate (Havlicek, 1968; Gates &

Bradshaw, 1974; Gates, Bradshaw & Nettleton, 1974; Finney 1997; Moelants, De-

mey & Leman, 2009). Another relevant effect is the increase in timing variability

(Pfordresher & Palmer, 2002; Pfordresher, 2003; Moelants, Demey & Leman, 2009).

As noticed in the previous paragraph, in some studies timing disruption and increase

in error rates may be correlated since repetitions and insertions would cause longer

performances. Nevertheless, researches in which error affected performances are re-

37

moved from the analysis show that timing aspects of the performance are disrupted

anyway (Pfordresher & Palmer, 2002; Pfordresher, 2003; Pfordresher & Benitez,

2007). For what concerns the dependence of timing disruption on the amount of de-

lay, once again the relative position of feedback onsets with respect to produced IOIs

predicts disruption better than their absolute position (Pfordresher & Benitez, 2007).

In general, timing disruption increases with the temporal separation of feedback on-

sets from produced actions, it reaches its maximum when feedback onsets approach

the time of the subsequent produced action, and significantly decreases when feed-

back onsets and produced actions are in phase synchrony. Pfordresher & Palmer

(2002) indicated a reduced impairment also for .5 phase ratio (anti-phase coordina-

tion), but this result was not confirmed by other experiments (Pfordresher, 2003;

Pfordresher & Benitez, 2007).

2.2.3 Dynamics

All research on DAF and music playing which considered dynamics aspects of per-

formance (Havlicek, 1968; Finney, 1997; Moelants, Demey & Leman, 2009) re-

ported significant increase in the playing intensity.

2.2.4 Expressivity

At the current state of the art, no research has explicitly focused on the effects of

DAF on expressivity. However, it can be argued that expressiveness is also debili-

tated by DAF, at least because it can be considered as a sum of timing and dynamics

deviations from a “flat” performance (constant tempo and intensity), and both timing

and dynamics are shown to be disrupted by DAF.

2.2.5 Body movement

The only study focusing on body movement (Moelants, Demey & Leman, 2009) re-

ports a significant increase in head movement in three out of four of the tested musi-

cal pieces.

38

39

CHAPTER 3 - METHOD

In this Chapter, I present the experimental method followed in the investigation. 16

keyboard players were asked to play a piece on an electronic piano under three dif-

ferent auditory conditions: first under a normal auditory feedback condition, then

without any auditory feedback, in the end with a feedback delay of 300 ms. Execu-

tions were recorded in MIDI format. Body movement was recorded with two sen-

sors, one positioned on the forehead and the other on the chest of the subjects. All the

experiments took place at IPEM in Ghent, Belgium. Figure 3.1 shows a participant

during the experiment.

Figure 3.1 A screenshot from the video recording of the experiment.

40

3.1 Subjects

16 adult musicians, numbered from 1 to 16, took part in the experiment. They were

given some University Cinema tickets as an incentive. The data from subject 1 were

discarded because MIDI and sensors data relative to his performances were not accu-

rately synchronized. Therefore, only the data of subjects from 2 to 16 were exam-

ined. 8 subjects were male, 7 were female. The average age was 30.6 years, ranging

from 18 to 71. It is possible to divide participants into two subgroups, depending on

whether they played the organ or not. A first group of eight participants (subjects

from 2 to 9) was composed of pianists recruited at the Conservatory and at the Uni-

versity of Ghent, Belgium. None of them had experience with organ playing. The

remaining seven participants (subjects from 10 to 16) were organists from the Ghent

and the Antwerp communities. For three subjects of this second group, organ was not

the only keyboard instrument played: subject 13 and 15 were also pianists, and sub-

ject 16 was also a harpsichord player. Only three subjects, within this group, had a

long-lasting experience with organ: subject 10, 11 and 12. Instead, subjects from 13

to 16 had studied the organ for less than three years. The average years of keyboard

studying among all the 15 subjects, regardless of the kind of keyboard instrument

played, was 18.1 years, ranging from 2 to 60. These data are represented in Table

3.1.

41

Subject Age Sex Piano Pipe Organ

Harpsicord Keyboard Instruments

Years Years Years Years

2 35 M 28 28

3 19 F 10 10

4 22 M 9 9

5 20 M 12 12

6 26 F 16 16

7 23 F 15 15

8 31 M 11 11

9 23 F 12 12

10 51 M 25 25

11 71 M 60 60

12 43 M 31 31

13 18 F 7 1 7

14 27 M 2 2

15 28 F 20 2 20 16 22 F 2 13 13 Mean 30.6 18.1

St. Dev. 14.5 14.1

Table 3.1 Subjects’ ID, age, sex, and years of keyboard instruments studying.

3.2 Stimulus material

An excerpt of the well-known “Sarabande”, from the “Harpsichord Suite IV in D

Minor HWV 437” by George Friedrich Händel, served as stimulus material. The

sarabandes are slow and grand Baroque stylized dances in triple meter, very popular

in the European courts of the 17th and 18th century. They were characterized by ac-

cents on the second beat of each measure. The second and the third beats were often

tied, giving the dance a distinctive rhythm of quarter and halves notes in alternations.

Händel’s “Sarabande” was written around 1703-06 and was first published in 1733.

Originally composed for harpsichord, it can be played on the piano with excellent re-

sults, and transcriptions for organ are common too. The excerpt used as stimulus was

constituted by the first 16 measures of the piece, a period made of two parallel

phrases: the first phrase ends with a half cadence, while the second ends with a per-

fect authentic cadence. Every phrase can be divided in 4 sub-phrases of 2 measures

42

each, with a rhythm typical of the sarabandes of Händel’s early years (cf. Burrows,

1997): . The piece was notated in the key of D minor and featured a 3/2

meter. It had to be performed with both hands. The score given to the musician is

showed in Figure 3.2. The chosen tempo of the execution, 56 bpm per half note, was

also indicated.

Figure 3.2 The excerpt of the “Sarabande” by Händel used as stimulus material.

The total number of notes to play was 165, the total number of IOI 64. The maxi-

mum IOIs values were found in connection with quarter notes: in this cases, the

nominal IOI value at 56 bpm was 536 ms. The piece was chosen after a pilot experiment in which a skilled pianist executed

three pieces with different characteristics: besides the “Sarabande”, “Für Elise” and

“33 Veräderungen uber einen Walzer von A. Diabelli” by Ludwig van Beethoven.

“Sarabande” was chosen for various reasons. First, it is a Baroque piece for harpsi-

chord (sustain pedal is not needed), but commonly played on organ and piano too.

Therefore, every keyboard player can perform it regardless of the specific instrument

studied. The second reason is that the “Sarabande” is an easy piece to play, within

the capacity of every fairly skilled keyboard player. Third reason, from the analysis

of the pilot study “Sarabande” was found to be easier to analyze than the other

pieces. In fact, it is a slow and majestic piece, in which different chords and notes are

clearly detectable, so that it is straightforward to divide the piece in different sub-

units, referring to the score. For the same reason, note errors are less frequent and, if

present, easy to find. The fourth and last advantage of using the “Sarabande” is that

43

at the chosen tempo of 56 bpm the maximum nominal IOI is of 536 ms: considering

a delay time of 300 ms for the DAF condition (the delay time actually used), IOIs are

always longer than the delay time. Therefore, every feedback onset associated with a

produced event i is asynchronous with i but precedes the produced event 1i (see

Figure 3.3). This fact is important because the opposite situation, namely the feed-

back onset of a produced event i following the produced event 1i (IOIs shorter than

delay time), would have caused feedback to be altered not only in phase but also in

pitch. Pfordresher (2003, cf. Paragraph 1.3.2) demonstrated that DAF disruption in

the case of pitch shifts is different from disruption in case of phase shifts: while the

first primarily disrupts sequencing, the second primarily disrupts timing. If a faster

piece like “Für Elise” had been chosen, both kind of alterations would have been pre-

sent. In such circumstances disruption may reflect both feedback timing and pitch, in

this way complicating the analysis of the influencing factors. The choice of the slow

“Sarabande” as stimulus material permitted us to avoid this problem.

Figure 3.3 Relationships between produced onsets and feedback onsets within the

main rhythmic figure of the “Sarabande”, assuming a 56 bpm tempo

1071 1607 536

1071 1607 536

300 300 300 871 1407 236

536 (=)

0

Time (ms)

Planned IOIs

Produced Onsets / Feedback Onsets Intervals

Feedback Onsets

Produced Onsets

44

3.3 Conditions

Each subject performed the chosen piece under three different auditory condition:

normal feedback, silent feedback and delayed feedback (DAF). Under the DAF con-

dition, the sound of each keystroke was heard at a fixed delay time of 300 ms after

the absolute time of the event production. Being the piece not isochronous, the rela-

tionships between IOIs and delay time are variable (see Figure 3.3): at the chosen

tempo of 56 bpm, delay time corresponded to 56% of quarter notes IOIs, to 28% of

half notes IOIs, and to 19% of dotted half notes IOIs. The delay time was chosen af-

ter a basic statistical analysis of the data of the pilot experiment (cf. Paragraph 3.2),

in which the pianist tested the pieces with 8 different delay times: 90, 140, 190, 240,

290, 340, 490, and 890 ms. From the data analysis values around 300 ms (290, 340

ms) resulted to work well with the slow “Sarabande”. In fact, lower values had less

influence on the performance, whereas higher values caused the problem of the over-

lapping of phase shifts and pitch shifts discussed in the previous paragraph.

3.4 Equipment

The subjects performed the piece on a Yamaha P60 weighted-key digital keyboard,

which simulates the feel of an acoustic piano. The keyboard was connected to a per-

sonal computer via a M-Audio MIDI USB Interface. The computer used was a Fu-

jitsu Siemens laptop endowed with a AMD Sempron 3000+ processor clocked at 798

MHz and 992 MB of RAM. The audio card was an external Creative Sound Blaster

Audigy 2 NX. An electronic Korg MA-30 metronome was used to suggest subjects

the performance tempo. No sustain pedal was used. Under normal conditions, the

subjects heard the auditory feedback directly from the piano speakers. Under this cir-

cumstance, the selected sound patch was Grand Piano 1, a preset patch simulating a

standard acoustic piano timbre. The sound intensity was kept at the same level with

all the subjects. The silent condition was obtained just lowering the volume control

to zero (the feedback due to the physical key noise from the keyboard wasn’t taken

into consideration). The DAF condition was obtained writing a Pure Data patch (see

Figure 3.4) which synthesized the incoming MIDI notes after a settable time interval.

To set up the system and create such condition, the interval was first set equal to

45

zero: in this way it was possible to measure the intrinsic delay of the system formed

by piano, MIDI interface and computer audio card. Then, appropriate values were

added to this intrinsic delay, in order to obtain the desired delay times. The notes

synthesized by the audio card were played by two speakers positioned over the key-

board ones. The synthesizer sound timbre was chosen and balanced in order to be as

similar as possible to the Yamaha P60 Grand Piano 1. The volume was kept ap-

proximately equal to the volume of the piano speakers under normal conditions.

Keystroke MIDI data were collected using the same Pure Data patch used to generate

the delayed auditory feedback condition (Figure 3.4).

Figure 3.4 The Pure Data patch used to record MIDI data and to add the delay time

Movement data were detected through two MT Xsens sensors capable of calculating

the Tait–Bryan angles yaw, pitch and roll in real time, as well as outputting cali-

brated 3D linear acceleration; one sensor was positioned on the forehead of the sub-

46

jects, the other one on the chest. In Figure 3.5 a representation of the Tait-Bryan an-

gles relative to the human head movements is shown.

Figure 3.5 The Tait–Bryan angles yaw, pitch and roll relative to the human head

movements.

One more sensor was used to synchronize MIDI and sensors data. Data extraction

was performed with Matlab R2007b, whereas data analysis was performed with

Matlab R2007b and SPSS 15.0. All the experiments were recorded with a JVC MG-

30 digital camera.

3.5 Procedure

After the selection, each subject was given a few days to learn the score of the ex-

cerpt (Figure 3.2). At that moment no explanation was given about the specific ex-

periment to be conducted. Subjects were then tested individually. First, they were

given a pre-questionnaire to examine their musical training and background. Sensors

were then positioned on forehead and chest of the players using elastic bands. Before

starting the measurements, each player had the possibility of training some minutes,

with the musical notation present. Only in this training phase, a metronome beat in-

dicated the players the suggested tempo of 56 bpm. They weren’t asked to strictly

47

keep the tempo during the execution, but to use it just as a starting reference, and

then add all the timing expressiveness (accelerando, rallentando) they considered ap-

propriate. They were also told to avoid the use of non-written ornaments such as trills

or passing notes. Another indication given was not to stop in case of errors, but to

continue the execution until the end of the piece, trying to keep the desired execution

rate. After the training phase, the metronome was stopped, while the notation was

left available on the note-holder. The three auditory conditions (normal, silent and

delayed feedback) were executed in succession, with short breaks in between to con-

sent the necessary changing in the experimental apparatus. Each condition was de-

scribed to the subjects just before the start of the respective recordings. For each

condition, the start of the recordings was given pressing sharply the A0 key of the pi-

ano with a sensor. This action had the specific function of consenting the subsequent

finding of a temporal relationship between the MIDI and the sensors data streams:

during the data extraction phase, the A0 keystroke time recorded by the MIDI was

bound to the time of the vertical acceleration peak recorded by the sensor, thus ob-

taining a sufficiently precise temporal coordination between the two streams. In the

continuation of the thesis, I will refer to the sensor used to strike the A0 key as to the

coordination sensor. If a player stopped in the middle of a recording, the execution

under that condition was restarted from the beginning, and, in case, restarted again,

until a complete execution was recorded for each condition. After the measurements

phase, a post-questionnaire was given to subjects, to express their subjective impres-

sions about the experienced conditions. In particular, questions were asked about the

difficulty of the tasks, the emotions felt in the various conditions, the degree of ex-

pressiveness added to the piece, and the strategies adopted to cope with the delay.

48

49

CHAPTER 4 – DATA ANALYSIS

In this chapter, I describe the techniques adopted to extract the parameters to be ana-

lyzed, starting from the recorded data. A Matlab script was written for this purpose.

Such script first imported both MIDI and movement data, respectively from the .txt

files outputted by the Pure Data patch and from the .log files outputted by the MT

software. MIDI and movement data where then synchronized following the method

described in Subparagraph 4.2.1. Working on the imported data, the Matlab script

extracted all the parameters of relevance for the subsequent analysis. Some of these

parameters are calculated with reference to musical measures: such parameters are

tempo, average key velocity, and average intensity of movement (IoM) of head and

chest. The calculation of tempo and average key velocity is described in Paragraph

4.1, whereas calculation of IoM is explained in Subparagraph 4.2.2. For what con-

cerns the periodicity of the movement, the Fourier spectra of the head pitch signals

were calculated. The focus was placed on the spectral magnitude of three particularly

relevant frequencies: the frequencies corresponding to 1-beat, 1-measure, and 2-

measures periodicities. These spectral values are of interest because possible peaks in

correspondence to them would signify regular movements of the head in relation to

the metrical subdivision of the piece. The Fourier analysis is described more in detail

in Subparagraph 4.2.3.

4.1 MIDI data

The Matlab script automatically extracts from the MIDI data the first note of each

measure, to obtain a subdivision per measure of all the notes of the performance (see

Figure 4.1). For each measure, the first note was chosen as the first note played be-

tween the notes belonging to the nominal first chord of the measure. Subdivision per

measure was then checked manually, to correct the errors due to missing or wrong

notes.

50

Figure 4.1 MIDI piano-roll view of an execution of the Händel’s “Sarabande”. The

vertical dotted lines indicate the starting point of each measure.

Starting from such subdivision, average tempo (in bpm) and key velocity were calcu-

lated for each measure. Given a measure k, the average tempo for this measure was

calculated with the following formula, where )(ksp indicates the starting point (in s)

of the k-th measure relative to the start of the performance, and QPM indicates the

number of quarters per measure (in our case, QPM is constant and equal to 3):

60)()1(1)(

QPMkspkspktempo . (4.1)

The average key velocity of each measure was calculated as the average of the key

velocity of all the notes belonging to the measure.

51

4.2 Movement data

4.2.1 Synchronization of MIDI and movement data streams

As written in Paragraph 3.5, before every performance a striking of the A0 key with

the coordination sensor was used to relate MIDI data with movement data streams.

The timing of the peak in the vertical acceleration of the coordination sensor (Figure

4.2), approximately indicating the moment of contact between the key and the key-

bed (cf. Goebl & Palmer, 2008, case of pressed touch), was taken as starting point

for the movement recording, and bound together with the timing of the correspond-

ing keystroke recorded by the MIDI.

Figure 4.2 The peak in the vertical acceleration of the coordination sensor due to the

striking of the A0 key.

Having established in this way a temporal relationship between movement and MIDI

data, the starting points of each measure, extracted from the MIDI, were bound with

52

the corresponding samples in the movement data stream, so that a subdivision per

measure of the movement data was obtained (Figure 4.3).

Figure 4.3 Subdivision per measure of the head angles data stream. Dotted vertical

lines indicate the starting point of each measure.


Moving from the 3D linear acceleration data stream, the absolute value of the jerk

(i.e., the temporal derivative of the acceleration) was taken into consideration as a

measure of the intensity of the movement, IoM:

222

dtda

dtda

dtda

dtadjIoM zyx

, (4.2)

where j

is the jerk. IoM values were then averaged over each measure.

53

4.2.3 Head pitch periodicity

For what concerns the movement angles, only the pitch of the head has been taken

into consideration during the analysis. The reason for this choice is that normally up-

down movements (pitch) of the player’s head are more related to the time-keeping

process and to the transmission of expressivity than left-right head movements (yaw)

and lateral head oscillations (roll). This point was experimentally confirmed by the

fact that head pitch shows much clear periodicities than yaw and roll, and that these

periodicities are more related to the subdivision of the music in measures and beats

(e.g., see Figure 4.3, in which pitch exhibits a periodicity related to measures). For

analogue reasons, chest angles were not considered. Before proceeding with the Fou-

rier analysis, some manipulations were done on the original head pitch signals. First,

head pitch data were resampled with linear interpolation so that each measure of each

performance contained the same number of samples. Second, signals were averaged

to zero, to cancel the 0 Hz components of their Fourier spectra. Third, signals were

zero-padded using a zero-padding factor (zpf) of 4, which was found to be the mini-

mum zpf needed to obtain spectral values in the three frequencies of interest for our

investigation. It has to be noticed that, thanks to the precise subdivision per measure

operated on each performance (cf. Paragraph 4.1), as well as to the fact that after the

resampling each measure has the same number of head pitch samples, calculated fre-

quencies for 1-measure and 2-measures are directly proportional to the actual meas-

ure frequency of performances. On the other hand, beat frequency is calculated di-

viding the 1-measure frequency for QPM, therefore resulting in an approximation of

the actual beat frequency in the performances. 4-zero-padding resampled head pitch

signals were then transformed with a FFT algorithm. The magnitude of the Fourier

transforms was normalized so that the sum of all the samples of each transform was

one. For the three relevant frequencies, a weight w was calculated as the ratio be-

tween the spectral magnitude for that frequency and the overall standard deviation of

the spectrum. An example of the resulting Fourier spectrum is shown in Figure 4.4.

54

Figure 4.4 Normalized FFT of the resampled head pitch signal of Figure 4.3. The

highest peak is in correspondence to the 1-measure frequency. The weight (w) of the

relevant frequencies indicates the ratio between the signal magnitude for such fre-

quencies and the overall standard deviation.

55

CHAPTER 5 – RESULTS

In the present chapter, I will proceed with the statistical analysis of the effect of si-

lent and delayed auditory feedback on the extracted parameters, first considering, for

each experimental condition and each parameter, the data from all the subjects as a

single distribution. Although there are individual differences between pianists, and it

can be expected that the capability to cope with delay will depend on personal factors

(e.g. skills or education), it is assumed that most pianist will respond in a similar way

depending on the condition. The technique chosen to investigate this hypothesis is a

one-way analysis of variance (ANOVA). In Paragraph 5.1, the one-way ANOVA

will be performed on those parameters for which we extracted values per measure

(cf. Chapter 4): tempo, average key velocity, average head and chest IoM. In Para-

graph 5.2, for each of the expressive parameters of the performance (tempo and aver-

age velocity), the correlations between the various conditions will be calculated, to

check how similar these parameters result in each pair of corresponding measures. In

this way, we intend to investigate possible dissimilarities in the expressive content of

the performances in the different conditions. Another way to look for different ex-

pressive contents is to compare the average timing and dynamics profiles: this will be

done in Paragraph 5.3, only with a descriptive purpose, due to the small sizes of the

considered distributions. Finally, in Paragraph 5.4, individual results will be summer-

ized. Subparagraph 5.4.1 will focus on tempo and velocity, Subparagraph 5.4.2 on

the intensity of movement, while Subparagraph 5.4.3 will report the results of the pe-

riodicity analysis.

5.1 One-way ANOVA of tempo, dynamics, and intensity of move-

ment against the experimental condition as factor

5.1.1 Exploratory data analysis

In Figure 5.1, the boxplots of tempo, average key velocity, head and chest IoM by

experimental condition are shown. The graphics indicate that, in the case of tempo,

velocity, and chest IoM, the normality of the distributions can be accepted, although

56

with some uncertainty. This is not the case, however, of the movement of the head:

especially in the delayed case, a large amount of extreme values and outliers is pre-

sent. After an inspection of the individual sequence plots, it was found that there are

3 participants (subjects 8, 11 and 12) who moved a lot more their head in the delayed

case, in comparison with the other participants (Figure 5.2). The effect of these 3

subjects on the distributions can be seen when they are excluded from the analysis

(Figure 5.3).

a b

c d

Figure 5.1 Boxplots of tempo (a), average velocity (b), head IoM (c) and chest IoM

(d) by experimental condition (from left to right: normal, silent delayed).

57

Figure 5.2 Example of difference in the head movement (left subject 5, right subject

8).

a b

c d

Figure 5.3 Boxplots of tempo (a), average velocity (b), head IoM (c) and chest IoM

(d) by experimental condition (from left to right: normal, silent delayed), without

subjects 8, 11 and 12.

In the following sections, a one-way ANOVA is used to look for effects of the ex-

perimental condition on the measured variables. In order to have an idea about the

58

impact of subject 8, 11 and 12, the ANOVA is performed on the whole dataset and

on the dataset excluding subjects 8, 11 and 12.

5.1.2 Homogeneity of variances

In order to choose the correct settings for the analysis, a Levene's test of homogene-

ity of variances is performed (see Table 5.1). All significances are lower than 0.05

for both analyses. This means that homogeneity of variance cannot be assumed.

Therefore, a Tamhane's T2 post hoc test instead of a Tukey test is used in the

ANOVA analysis (the Tamhane’s test uses the Welch procedure for determining de-

grees of freedom for the standard error of the contrast; this test is based on the Stu-

dent’s t distribution and the Sidak procedure is applied to find the alpha level).

Test of Homogeneity of Variances - All Cases

Levene Statistic df1 df2 Sig.

Tempo 6,528 2 672 ,002

Average velocity 4,855 2 672 ,008

HD_Iom 42,367 2 672 ,000

CH_Iom 12,334 2 672 ,000

Test of Homogeneity of Variances - 8, 11 and 12 excluded

Levene Statistic df1 df2 Sig.

Tempo 4,366 2 537 ,013

Average velocity 8,778 2 537 ,000

HD_Iom 3,563 2 537 ,029

CH_Iom 9,847 2 537 ,000

Table 5.1 Levene’s tests.

5.1.3 ANOVA table

Table 5.2 and 5.3 are respectively the ANOVA tables for the all cases and for the se-

lected cases (subject 8, 11 and 12 excluded). The low significance values provide

evidence of significant differences between the experiments for both analyses.

59

Sum of Squares df Mean Square F Sig.

Between Groups 5818,287 2 2909,143 48,256 ,000

Within Groups 40511,627 672 60,285 Tempo

Total 46329,914 674

Between Groups 11524,866 2 5762,433 49,700 ,000

Within Groups 77915,005 672 115,945 Average velocity

Total 89439,871 674

Between Groups 1,178 2 ,589 52,971 ,000

Within Groups 7,473 672 ,011 Head_Iom

Total 8,651 674

Between Groups ,259 2 ,130 25,652 ,000

Within Groups 3,398 672 ,005 Chest_Iom

Total 3,657 674

Table 5.2 Anova table comprising all cases.

Sum of Squares df Mean Square F Sig.

Between Groups 5943,931 2 2971,966 44,601 ,000

Within Groups 35782,539 537 66,634 Tempo

Total 41726,470 539

Between Groups 6736,527 2 3368,264 34,066 ,000

Within Groups 53095,874 537 98,875 Average velocity

Total 59832,401 539

Between Groups ,131 2 ,066 13,279 ,000

Within Groups 2,652 537 ,005 Head_Iom

Total 2,783 539

Between Groups ,122 2 ,061 14,060 ,000

Within Groups 2,323 537 ,004 Chest_Iom

Total 2,444 539

Table 5.3 Anova Table for the selected data (subject 8, 11 and 12 excluded).

60

Some explanations will be now given about the values present in the tables. The total

sum of squares ( totSS ) is defined as the sum of the squares between groups ( bgSS )

and the sum of the squares within groups ( wgSS ):

wgbgtot SSSSSS , (5.1)

from which

k

i

n

jiij

k

iii

k

i

n

jij

ii

yyyynyy1 1

2

1

2

1 1

2 , (5.2)

with k the number of trials (or conditions), here 3, and in the number of measure-

ments for condition i, here 2251515 for all the conditions. The degrees of free-

dom (df) for a total number of measurements N are calculated as follows:

1 Ndftot (here 1675 ), (5.3)

1 kdfbg (here 13 ), (5.4)

kNdfwg (here 3675 ). (5.5)

The mean square (MS) values are calculated as the sum of squares divided by the

corresponding df, and are a measure of the variance. For example, in the case of

tempo and considering the all subjects (Table 5.2), we can see that the largest amount

of variance is explained by the condition. The valueF is then the ratio between the

mean square between groups and the mean square within groups:

wgbg MSMSvalueF . (5.6)

Again considering the case of tempo and all data, by way of example, the valueF

is 3.483.601.2909 . In this case, the critical valueF critF at 05.0 , with

)672,2(df , is equal to 3: critFF , therefore the null hypothesis that the variance is

due to chance is rejected in favour of the hypothesis that there is a difference among

the means in the 3 conditions.

The ANOVA tables provide evidence for an effect of condition on the mean values

of the investigated variables. In the next subparagraph, in order to explain which

conditions differ from others, post-hoc tests are used.

61

5.1.4 Tamhane's T2 post hoc test

Table 5.4 shows the results of the Tamhane’s post hoc test for the all data. In the case

of tempo, the table shows that the effect of delay is negative, meaning that delay

causes most players to decrease in tempo. The velocity, on the contrary, reveals a

positive effect: due to the delay a higher average velocity occurs. The intensity of

movement variables shows significant differences between all 3 experiments.

62

95% Conf. Interval Dependent

Variable (I) Experim. (J) Experim.

Mean Difference

(I-J) Std. Error Sig. Low. Bound Up. Bound

No audio -,72280 ,68454 ,645 -2,3638 ,9182 Normal

Delayed 5,83511* ,71823 ,000 4,1131 7,5571

Normal ,72280 ,68454 ,645 -,9182 2,3638 No audio

Delayed 6,55791* ,78940 ,000 4,6659 8,4499

Normal -5,83511* ,71823 ,000 -7,5571 -4,1131

Tempo

Delayed

No audio -6,55791* ,78940 ,000 -8,4499 -4,6659

No audio -1,32733 1,04678 ,498 -3,8363 1,1816 Normal

Delayed -9,35338* 1,02917 ,000 -11,8202 -6,8866

Normal 1,32733 1,04678 ,498 -1,1816 3,8363 No audio

Delayed -8,02604* ,96795 ,000 -10,3459 -5,7062

Normal 9,35338* 1,02917 ,000 6,8866 11,8202

Average

velocity

Delayed

No audio 8,02604* ,96795 ,000 5,7062 10,3459

No audio ,027238* ,006643 ,000 ,01132 ,04316 Normal

Delayed -,071808* ,011396 ,000 -,09916 -,04446

Normal -,027238* ,006643 ,000 -,04316 -,01132 No audio

Delayed -,099046* ,011070 ,000 -,12563 -,07247

Normal ,071808* ,011396 ,000 ,04446 ,09916

HD_Iom

Delayed

No audio ,099046* ,011070 ,000 ,07247 ,12563

No audio ,020980* ,006103 ,002 ,00635 ,03561 Normal

Delayed -,026916* ,007373 ,001 -,04459 -,00924

Normal -,020980* ,006103 ,002 -,03561 -,00635 No audio

Delayed -,047895* ,006574 ,000 -,06366 -,03213

Normal ,026916* ,007373 ,001 ,00924 ,04459

CH_Iom

Delayed

No audio ,047895* ,006574 ,000 ,03213 ,06366

*. The mean difference is significant at the 0.05 level.

Table 5.4 Tamhane's T2 post-hoc test, all data.

When excluding subjects 8, 11 and 12, the conclusions for tempo and average veloci-

ties remain the same. On the other hand, it is found that the intensity of movement is

different in the case of no feedback (the subjects move less in the absence of feed-

63

back) and that there is no significant difference between the normal and delayed con-

dition.

95% Confidence Interval Dependent

Variable (I) Experim. (J) Experim.

Mean Difference

(I-J) Std. Error Sig. Low. Bound Up. Bound

No audio -,70022 ,80969 ,771 -2,6432 1,2427 Normal

Delayed 6,66167* ,83962 ,000 4,6466 8,6767

Normal ,70022 ,80969 ,771 -1,2427 2,6432 No audio

Delayed 7,36189* ,92767 ,000 5,1364 9,5874

Normal -6,66167* ,83962 ,000 -8,6767 -4,6466

Tempo

Delayed

No audio -7,36189* ,92767 ,000 -9,5874 -5,1364

No audio -1,61622 1,10948 ,377 -4,2782 1,0457 Normal

Delayed -8,16872* 1,06624 ,000 -10,7274 -5,6100

Normal 1,61622 1,10948 ,377 -1,0457 4,2782 No audio

Delayed -6,55250* ,96334 ,000 -8,8636 -4,2414

Normal 8,16872* 1,06624 ,000 5,6100 10,7274

Average

velocity

Delayed

No audio 6,55250* ,96334 ,000 4,2414 8,8636

No audio ,027049* ,007446 ,001 ,00918 ,04491 Normal

Delayed -,009802 ,007763 ,502 -,02843 ,00882

Normal -,027049* ,007446 ,001 -,04491 -,00918 No audio

Delayed -,036852* ,006993 ,000 -,05363 -,02007

Normal ,009802 ,007763 ,502 -,00882 ,02843

HD_Iom

Delayed

No audio ,036852* ,006993 ,000 ,02007 ,05363

No audio ,023426* ,006276 ,001 ,00836 ,03849 Normal

Delayed -,012821 ,007641 ,257 -,03115 ,00551

Normal -,023426* ,006276 ,001 -,03849 -,00836 No audio

Delayed -,036246* ,006811 ,000 -,05260 -,01989

Normal ,012821 ,007641 ,257 -,00551 ,03115

CH_Iom

Delayed

No audio ,036246* ,006811 ,000 ,01989 ,05260

*. The mean difference is significant at the 0.05 level.

Table 5.5 Tamhane’s posthoc test, selected data (subjects 8, 11 and 12 excluded).

64

5.1.5 Homogeneous subset tables

The homogeneous subset tables enable to divide factors into subsets, resulting in an

alternative way to look for differences or equalities between factors. By way of ex-

ample, for what concerns tempo and all data, it can be seen that the delayed condition

differs from the normal and the silent, which instead belong to the same subset. The

conclusions which can be drawn from these tables correspond, as expected, with

those of the Tamhane’s table.

Tempo

Subset for alpha = 0.05

Experiment N 1 2

Delayed 225 56,7759

Normal 225 62,6110

No audio 225 63,3338

Sig. 1,000 ,585

Average Velocity


Experiment N 1 2

Normal 225 58,4158

No audio 225 59,7431

Delayed 225 67,7692

Sig. ,391 1,000

Head_Intensity of movement


Experiment N 1 2 3

No audio 225 ,16033

Normal 225 ,18757

Delayed 225 ,25938

Sig. 1,000 1,000 1,000

Chest Intensity of movement


Experiment N 1 2 3

No audio 225 ,18425

Normal 225 ,20523

Delayed 225 ,23214

Sig. 1,000 1,000 1,000

Table 5.6 Homogeneous subset table, all data.

65

Tempo

Subset for alpha =

0.05

Experiment N 1 2

Delayed 180 56,3690

Normal 180 63,0307

No audio 180 63,7309

Sig. 1,000 ,695

Average velocity

Subset for alpha =

0.05

Experiment N 1 2

Normal 180 58,7731

No audio 180 60,3893

Delayed 180 66,9418

Sig. ,272 1,000

Head Intensity of Movement

Subset for alpha =

0.05

Experiment N 1 2

No audio 180 ,16548

Normal 180 ,19253

Delayed 180 ,20233

Sig. 1,000 ,383

Chest Intensity of movement

Subset for alpha =

0.05

Experiment N 1 2

No audio 180 ,19105

Normal 180 ,21447

Delayed 180 ,22729

Sig. 1,000 ,155

Table 5.7 Homogeneous subset table, selected data (subjects 8, 11 and 12 excluded).

5.1.6 Means plots

The means plots visualize the results of the analysis showing the effects of the 3 fac-

tors on the 4 variables.

66

Figure 5.4 Marginal means plots, all data. Top left: tempo; top right: velocity; bot-

tom left: head IoM; bottom right: chest IoM.

67

Figure 5.5 Marginal means plots, selected data (subjects 8, 11 and 12 excluded). Top

left: tempo; top right: velocity; bottom left: head IoM; bottom right: chest IoM.

5.1.7 Summary of the one-way ANOVA results

1) The ANOVA reveals that there is a significant effect on tempo and average ve-

locity as a result of the effect of delay, considering all participants’ data. In par-

ticular, delay decreases the tempo and increases the average velocity.

2) The intensity of movement is more complex: 3 of the 15 participants have a

different behavior and move more intensively in the delayed context.

5.2 Correlations

Now, the Pearson’s correlation coefficients r between the values of the expressive

parameters of the performance, tempo and average velocity per measure, in the three

experimental conditions, will be calculated. As in the previous paragraph, the data of

all the subjects, for a given parameter and a certain condition, are considered as a sin-

68

gle distribution: we saw in Subparagraph 5.1.1 that for such distributions normality

can be accepted. Results are shown in Table 5.8 and 5.9.

Correlations - Tempo

Tmp_N Tmp_S Tmp_D

Pearson Correlation 1 ,848(**) ,621(**) Sig. (2-tailed) ,000 ,000

Tmp_N

N 225 225 225 Pearson Correlation ,848(**) 1 ,574(**) Sig. (2-tailed) ,000 ,000

Tmp_S

N 225 225 225 Pearson Correlation ,621(**) ,574(**) 1 Sig. (2-tailed) ,000 ,000

Tmp_D

N 225 225 225 ** Correlation is significant at the 0.01 level (2-tailed).

Table 5.8 Correlations between the values of tempo in the three experimental condi-

tions (N = normal, S = silent, D = delayed).

Correlations – Average Velocity

Avel_N Avel_S Avel_D

Pearson Correlation 1 ,851(**) ,689(**) Sig. (2-tailed) ,000 ,000

Avel_N

N 225 225 225 Pearson Correlation ,851(**) 1 ,743(**) Sig. (2-tailed) ,000 ,000

Avel_S

N 225 225 225 Pearson Correlation ,689(**) ,743(**) 1 Sig. (2-tailed) ,000 ,000

Avel_D

N 225 225 225 ** Correlation is significant at the 0.01 level (2-tailed).

Table 5.9 Correlations between the values of average velocity in the three experi-

mental conditions (N = normal, S = silent, D = delayed).

Although all the parameters are significantly correlated ( 01.0Sig ), we can see that

the correlations between the values in the normal and the delayed conditions (tempo:

621.0r ; velocity: 689.0r ) are lower than the correlations between the values in

the normal and the silent (tempo: 848.0r ; velocity: 851.0r ). Another way to

look at this differences is to draw the overlay scatter plots (Figure 5.6 and 5.7).

69

Figure 5.6 The relationships between tempo per measure in the normal and silent

conditions (green) vs the relationships between tempo per measure in the normal and

delayed conditions (blue), with best-fit lines. Normal condition values are repre-

sented by the abscissas.

70

Figure 5.7 The relationships between average velocity per measure in the normal

and silent conditions (green) vs the relationships between average velocity per meas-

ure in the normal and delayed conditions (blue), with best-fit lines. Normal condition

values are represented by the abscissas.

5.3 Timing and dynamics profiles

Now, the average timing and dynamics profiles in the three experimental conditions

will be considered. With timing profile we refer to the variation in tempo as a func-

tion of measure, whereas with dynamics profile we refer to the variation in average

velocity as a function of measure. For a given condition, the profiles are calculated

averaging each measure data over all the subjects. Before proceeding, it must be

taken into account that, in our case, the number of samples for each measure-

condition pair is low (15). Therefore, these statistics will have only descriptive pur-

poses. In Figure 5.8 the average timing and dynamics profiles are shown.

71

Figure 5.8 The average timing (left) and dynamics (right) profiles, in the three con-

ditions.

The results show that, while the average profiles of the normal condition are kept in

the silent condition, they are disrupted in the delayed condition. In particular, a pro-

gressive slowing of production rate is evident in the timing profile, causing a re-

markable deterioration in the timing shape of the performance. For what concerns the

key velocity, although the dynamics shape results similar in all the three conditions,

in the delayed it is higher than in the others by a noteworthy constant factor. In Fig-

ure 5.9, the differences between the profiles in the normal and delayed conditions are

shown with error bars indicating the 95% confidence intervals. It can be noticed how,

in the last four measures of the timing profile, the error bars of the two conditions are

not overlapping. In the dynamics profile, instead, the error bars do not overlap only

in one measure (the 9th).

72

Figure 5.9 The average timing (left) and dynamics (right) profiles, in the normal

(blue) and delayed (green) conditions. Error bars indicate the 95% confidence inter-

vals.

5.4 Individual results

Now, some individual subjects results will be provided. As for the timing and dy-

namics profiles, the amounts of considered samples are low (15), therefore these sta-

tistics have only a descriptive aim. The complete dataset for each subject can be

found in graphical form in Appendix 1.

5.4.1 Tempo and velocity

Table 5.10 presents the individual subjects means and standard deviations for the ex-

pressive parameters (tempo and velocity) in the three conditions. Moreover, the Pear-

son’s correlation coefficients r between such parameters are shown for the two pairs

of conditions, normal-silent and normal-delayed. In Figure 5.10, 5.11, and 5.12, the

values of Table 5.10 are presented in graphical form.

73

TEMPO VELOCITY Subject Condition

Mean Std. Dev. r Mean Std. Dev. r Normal 55,64 2,48 53,05 13,65 Silent 55,04 1,66 ,705 59,74 9,21 ,821 2 Delay 50,52 2,64 -,078 69,75 5,13 ,643 Normal 62,04 2,26 46,67 5,58 Silent 65,12 2,09 ,719 46,36 5,48 ,881 3 Delay 60,10 4,06 ,160 54,70 5,34 ,671 Normal 61,75 1,91 64,02 8,56 Silent 66,03 1,27 ,027 59,24 7,94 ,660 4 Delay 47,78 6,21 ,716 72,01 6,79 ,668 Normal 64,13 3,79 43,97 7,02 Silent 67,25 3,34 ,868 51,13 6,84 ,867 5 Delay 63,64 2,42 ,265 63,56 8,03 ,848 Normal 64,79 2,67 55,14 7,02 Silent 69,88 3,17 ,580 63,52 5,44 ,738 6 Delay 62,90 6,75 ,230 77,53 3,43 ,631 Normal 63,50 1,90 49,72 4,67 Silent 66,49 1,70 ,500 54,79 3,76 ,788 7 Delay 57,56 2,53 -,078 54,98 3,66 ,749 Normal 65,65 3,41 58,48 7,01 Silent 68,33 3,42 ,827 53,34 5,52 ,799 8 Delay 58,33 4,21 ,130 71,60 2,23 ,171 Normal 67,27 3,79 54,14 4,64 Silent 70,89 4,67 ,511 50,76 5,21 ,683 9 Delay 54,79 7,29 ,566 59,33 4,83 ,532 Normal 69,76 3,18 76,48 2,75 Silent 67,27 2,83 ,639 74,36 2,95 -,125 10 Delay 62,89 4,07 ,744 76,51 2,21 ,074 Normal 57,47 3,04 69,35 9,07 Silent 58,41 2,91 ,833 72,43 9,04 ,806 11 Delay 51,23 2,14 ,303 86,28 7,91 ,893 Normal 59,68 2,09 43,12 4,87 Silent 58,50 1,79 -,019 45,71 5,58 ,766 12 Delay 65,65 3,91 ,375 55,36 6,09 ,753 Normal 73,12 3,06 69,08 2,71 Silent 74,77 2,37 ,626 66,73 2,68 ,292 13 Delay 65,77 5,94 ,082 72,09 3,39 ,260 Normal 49,43 3,31 66,47 4,20 Silent 44,99 6,87 ,042 70,63 2,09 ,063 14 Delay 37,24 6,47 -,030 66,38 5,18 ,431 Normal 66,08 4,19 62,73 5,86 Silent 61,21 3,72 ,803 63,47 3,58 ,752 15 Delay 56,62 2,31 -,068 68,23 3,73 ,315 Normal 58,86 1,82 63,81 4,32 Silent 55,82 2,19 ,765 63,95 3,35 ,536 16 Delay 56,62 2,31 ,244 68,23 3,73 ,031

Table 5.10 Individual subjects means and standard deviations of tempo and velocity

in the thee experimental conditions. In the r-labelled column, the Pearson’s correla-

tion coefficients r between such parameters are reported for the normal-silent (Silent-

74

labbelled row) and normal-delayed (Delay-labelled row) condition pairs. The high-

lights evidence the results in the delayed condition, in comparison with the normal

one: blue highlights indicate similarity between the values or high correlations, red

highlights remark bad matches or low correlations, whereas yellow highlights refer

to peculiar results.

Figure 5.10 Tempo: individual subjects means (left) and standard deviations (right)

in the three experimental conditions.

Considering the means of tempo in Figure 5.10, it can be noticed that some subjects

(3, 5, 6, 16) were able to keep approximately the same means in the normal and in

the delayed conditions. The difference between their means in the two conditions is

lower than 3 bmp. In general, however, most of the subjects show lower means val-

ues in the delayed condition, with the extreme cases of subjects 4, 9, and 14, for

which the difference between the means is higher than 10 bmp. A remarkable excep-

tion is given by subject 12, who is the only one who played faster in the delayed con-

dition. For what concerns the standard deviations, individual results are separable in

two groups, with subjects 2, 5, 7, 8, 10, 11 and 16 keeping similar values both under

the normal and in the delayed conditions, whereas subjects 3, 4, 6, 9, 12, 13, 14, and

15 show higher values in the delayed case. In particular, extreme cases in the two

groups are subjects 2, 7, 8, 10, and 11 (difference lower than 1) for the first, and sub-

jects 4, 6, 9, and 14 (difference higher than 3) for the second.

75

Figure 5.11 Average velocity: individual subjects means (left) and standard devia-

tions (right) in the three experimental conditions.

The individual mean values of key velocity (Figure 5.11 left) show that almost each

individual played louder with DAF than in the normal condition, with the exceptions

of subjects 10 and 14. For six subjects (2, 5, 6, 8, 11, 12), the difference between the

two conditions was really marked (more than 10 points). For what concerns the stan-

dard deviations, three subjects (2, 6, 8) show a much lower variability in the delayed

condition in comparison to the normal (the difference in standard deviations is higher

than 3). It is peculiar the case of subject 2, who had a very high dynamics variability

in the normal performance, much lower in the delayed condition (13.6 versus 5.1).

On the other hand, five subjects (3, 9, 10, 13, 14) have almost identical dynamics

variability (difference in standard deviation less than 1).

76

Figure 5.12 Individual subjects correlations (blue: normal-silent; green: normal-

delayed) for tempo (left) and velocity (right).

For what concerns the correlations in tempo between the normal and the delayed

cases (Figure 5.12 left green), only three subjects (4, 9, 10) show values higher than

0.5. The other subjects present low correlations values, with four subjects (2, 7, 14,

15) having negative correlations. Considering now the correlations between the nor-

mal and the delayed case, as for velocity (Figure 5.12 right green), the individual re-

sults are higher than those for tempo, with nine subjects having values superior to

0.5. Two of them (subjects 5 and 11), in particular, present correlations higher than

0.8.


In Figure 5.13, the means of head and chest IoM per subject are shown. The high in-

crease in the head movement for subject 8 and 12 in the delayed condition can be no-

ticed (left).

77

Figure 5.13 Individual subjects means of head (left) and chest (right) IoM in the

three experimental conditions.

5.4.3 Periodicity of the head pitch movement

For what concerns the periodicity of the head pitch, the calculated weights w for the

three frequencies of interest are reported in Table 5.11. The red highlights indicate

very significant peaks ( 20w ), whereas the yellow highlights indicate medium

range values ( 2010 w ). All individual subjects head pitch graphics can be seen

in Appendix 1, together with the plots of the corresponding resampled signals FFTs.

78

Subject Period.

Normal Silent Delay

Subject Period.

Normal Silent Delay 2-bars 14,6 20,4 7,9 2-bars 14,1 20,6 8,2 1-bar 3,7 7,3 25,3 1-bar 12,2 21,4 18,1 2 1-beat 0,3 0,2 2,1

10 1-beat 1,0 1,0 1,6

2-bars 4,5 5,5 12,9 2-bars 8,0 4,2 7,8 1-bar 7,5 6,4 13,6 1-bar 24,6 25,3 28,2 3 1-beat 3,2 1,6 3,3

11 1-beat 2,0 1,8 4,1


12 1-beat 2,1 11,0 34,1


13 1-beat 5,2 3,2 3,3


14 1-beat 5,4 4,7 4,6


15 1-beat 1,7 1,1 2,2


16 1-beat 1,0 0,5 1,8

2-bars 17,8 25,7 17,4 1-bar 18,2 14,0 17,9 9 1-beat 1,0 0,4 1,1

Table 5.11 Weights w of the spectral magnitude for the frequencies related to 2-

measures, 1-measure, and 1-beat. Red highlights indicate the very relevant peaks

( 20w ), whereas yellow highlights indicate medium range values ( 2010 w ).

A consideration of the weights in Table 5.11 and of the graphics of Appendix 1

makes clear that most subjects changed qualitatively or quantitatively the periodicity

of their head pitch in the different conditions. The Table 5.12 tries to point out these

changes when comparing the delayed case with the normal.

79

Subject Periodicity Body as a reference under DAF

2-bars 1-bar 1-beat (questionnaires)

2

3

x

4 =

x

5

x

6 =

x

7

= x

8

x

9 = =

10

11 =

12

x

13

x (trunk)

14

= =

15

16

Table 5.12 The changes in the individual head pitch periodicities when passing from

the normal condition to the delayed one. Vertical arrows indicate diminu-

tions/increases in the corresponding weights, horizontal arrows indicate changes in

the main peaks, whereas equal signs signal stationary situations. Thick arrows indi-

cate drastic changes. Cells are left blank when both conditions show no periodicity

for the correspondent frequency. On the right column, subjects who reported the use

of the body as a reference under DAF are marked.

Although the results show a great variability due to individual differences, two trends

seem to emerge in the delayed case: a tendency to reduce the relation between the

80

head pitch periodicity and the musical structures (subjects 5, 6, 7, 8, 13, and 16), and

a tendency to relate more with higher frequencies than with low ones, in comparison

to the normal condition (subjects 2, 4, 5, 7, 10, 11, and 12 ). Only three subjects (3,

14, 15) showed behaviours in opposition to these tendencies. Noteworthy is the be-

haviour of subjects 2 and 12, who drastically changed their periodicity in the delayed

case: subject 2 switched from very low-frequency movements (see Figure A1.2 in

Appendix 1) to a well-defined 1-measure periodicity (1-measure w passed from 3.7

in the normal case to 25.3 in the delayed), whereas subject 12 switched from a very

prominent 1-measure periodicity (31.5 in the normal case) to a equally prominent 1-

beat periodicity (34.1 in the delayed case). According to the questionnaires results,

subject 2 did not deliberately choose to change his movement under DAF, while

subject 12 did.

81

CHAPTER 6 – DISCUSSION AND CONCLUSIONS

In this Chapter, the experimental results will be discussed, with regards to each of the

considered aspects of music performance. Paragraph 6.1 will focus on timing, Para-

graph 6.2 on dynamics, Paragraph 6.3 on expressivity, and Paragraph 6.4 on body

movement. In Paragraph 6.5, some final conclusions will be drawn.

6.1 Timing

The ANOVA analysis of tempo confirmed the results found in literature. The ab-

sence of feedback does not have a significant effect on the distribution of the tempo

values, in comparison to normal conditions (the means are 62.6 bpm in the normal

condition and 63.3 bmp in the silent). This is in line with what found in previous

studies by Gates & Bradshaw (1974), Finney (1997), Repp (1999), Moelants, Demey

& Leman (2009). On the contrary, DAF has a strong effect on tempo, causing players

to significantly slow their production rate (the mean in the delayed case is 56.8 bmp).

This slowing effect of DAF, extensively studied with regard to speech tasks, has

been previously reported in music performance by Havlicek (1968), Gates & Brad-

shaw (1974), Gates, Bradshaw & Nettleton (1974), Finney (1997), and Moelants,

Demey & Leman (2009). Considering the analysis of the all subjects’ correlations

and timing profiles, performances under DAF resulted much more dissimilar with re-

spect to normal ones rather than performances in the absence of feedback. In particu-

lar, DAF performances tended to slow in tempo with the advance of the measures, to

the extent that, in the last four measures of the piece, the error bars of the normal and

delayed condition timing profiles do not overlap (cf. Figure 5.9) . This finding seems

to indicate an additive effect of timing disruption by DAF, so that the decrease in

tempo depends on the metrical positions. In other words, the players’ confusion

caused by DAF seems to increase measure after measure. However, the individual

timing profiles in Appendix 1 suggest that this effect may depend strongly on the in-

dividuals’ personal capabilities: although most subjects slowed down during the per-

formance, with subjects 4, 8 and 9 as extreme cases, other subjects (5, 10, 11, 15, 16)

did not show this tendency at all. In general, personal abilities seem to play an impor-

82

tant role in determining the effect of DAF on timing. While some subjects (5, 10, 11,

16) were capable of keeping the various aspects of timing almost at the same level

as in a normal performance, most of the performances degraded in one or more tim-

ing aspects. To conclude, noteworthy is the case of subject 12, who is the only one

who played faster in the DAF condition. As we will see in the Paragraph 6.4, this can

be interpreted as a consequence of a deliberately adopted embodied strategy.

6.2 Dynamics

The ANOVA analysis on the average key velocities provide further support to the lit-

erature results. The auditory feedback deprivation did not seriously affect the dynam-

ics of the performances (the mean in the silent condition is 59.7, against the 58.4

value in the normal condition), as previously reported by Finney (1997) and Repp

(1999). The DAF, instead, caused a significant increase in loudness (the mean value

in this case is 67.8), in accordance with the findings of Havlicek (1968), Finney

(1997), and Moelants, Demey & Leman (2009). A possible explanation for this fact

is that, under DAF conditions, the players attempted to rely more upon tactile feed-

back, which was not altered. Another possible account for this effect is that players,

having the impression that the notes were not “coming out”, tried to “force” them

playing louder. The all subjects’ correlations and dynamics profile show that DAF-

affected performances were less similar in the expressive dynamics to the normal

conditions performances as opposed to the silent condition ones. In particular, a

comparison of the dynamics profiles in the delayed and in the normal condition

shows that, although the average dynamics shapes were very similar, an almost con-

stant increase in loudness affected the values of each measure in the delayed case.

The individual subjects’ correlations between the normal and the delayed condition

were better than those regarding tempo, which may give support to the hypothesis

that the dynamics shapes suffer DAF less than the timing shapes (cf. Figure 5.8). As

in the case of tempo, however, a great individual variability seems to emerge.

83

6.3 Expressivity

The expressivity of the performances was more impaired in the case of DAF than in

the absence of auditory feedback. As for all subjects’ data, this is shown by the lower

correlations coefficients in the normal-delayed pair (Paragraph 5.2), as well as, in a

less significant but more impressive way, by the timing and the dynamics profiles

(Figure 5.9). The specific effects of DAF on the single parameters that shape an ex-

pressive piano performance, i.e. the expressive timing and the expressive dynamics,

are discussed in the previous paragraphs. An important support to the hypothesis that

DAF significantly disrupts the expressivity of the musical performance comes from

the analysis of the questionnaires. 13 subjects out of 15 affirmed they played expres-

sively in the normal condition. Considering the silent condition, 7 subjects stated

they could add the performance the same expressivity with respect to normal condi-

tions, 3 subjects reported they added more, and 5 subjects affirmed they could not

add the same expressivity. Under DAF, only 2 subjects (10 and 14) reported they

could play as expressively as in the normal condition, while the other 13 affirmed

they could not.

6.4 Body movement

The ANOVA analysis of the intensity of movement provide two results. First, the ab-

sence of auditory feedback caused subjects to move significantly less than in a nor-

mal condition. Second, in the DAF condition three subjects (8, 11, and 12, cf. Figure

5.13 left) moved their head much more than in the normal condition, causing the all

subjects’ values of the head IoM to deviate from normality: when they are excluded

from the analysis, the ANOVA reports no significant differences between the normal

and the delayed conditions, for both head and chest IoM. These results are in partial

contrast with the findings of Moelants, Demey & Leman, (2009), who, in a similar

study with 10 subjects, reported no differences in the amount of movement between

the normal and the silent condition in all the four pieces played, and a significant in-

crease of the head movement in the delayed condition in three out of four pieces. The

importance of the individual inclinations, pointed out in this study by the ANOVA,

may account for these different outcomes. The results concerning the periodicity of

84

the head pitch (Paragraph 5.4) seem to confirm the great variability in individual be-

haviours. However, although the dataset size does not permit to draw general conclu-

sions in this regard, the periodic movement of the players seems to be strongly influ-

enced by DAF. First, a tendency to reduce the periodicity components linked to the

musical structure seems to emerge. This fact can be interpreted as a disrupted effect

of DAF on body movement, in so far some subjects (5, 7, 8, and 13, cf. Table 5.12)

manifested such a tendency despite a deliberate attempt to use the body as a refer-

ence. The second noticeable tendency is that of relating more on high-frequencies

than on low, comparing with the normal condition. Two explanations may be taken

into account for these results. The first explanation relies upon the observation that,

under DAF, the expressiveness of the playing is highly impaired, as seen in the pre-

vious paragraph. In the DAF condition, most players found it difficult, if not even

impossible, to express the emotional contents of the music. In the “Sarabande”, as

seen in Paragraph 3.2, the musical sentences are based on sub-phrases of 2 measures

each, which may explain why, in the normal condition, players often moved with pe-

riodicities of this length. In the DAF condition, on the contrary, this natural link be-

tween body movement and expressivity seems to fail, resulting in a diminution in the

weights of the low-frequency periodicities. The second explanation relies upon the

fact that, under DAF, 8 subjects out of 15 explicitly tried to use their body as a refer-

ence (cf. Table 5.12), in order to contrast its disruptive effect on timing. This embod-

ied strategy may therefore account for the relative increase in the weight of the high-

frequency periodicities, more related to the beats. Noteworthingly, this embodied

strategy to cope with DAF was adopted by some subjects (2, 10, 11) who did not re-

port it as a conscious deliberation. Lastly, we focus on the behaviour of subject 12,

who, as we saw in Subparagraph 5.4.3, explicitly tried to keep the correct tempo

moving regularly his head at every beat (see Figure A1.22). This fact may account

for two results underlined before: his drastically increased IoM in the delayed condi-

tion (cf. Figure 5.13), and the fact that he is the only subject who played faster in the

delayed condition (cf. Figure 5.10). In conclusion, DAF seems to influence the pe-

riodicity of body movement both in relation to its disruptive effect on the expressiv-

ity and because it triggers embodied responses, conscious or not, that are functional

in contrasting its disruptive effects on timing.

85

6.5 Conclusions

Embodied strategies, such as using the head movement as a reference, seem to have a

strong potential for alleviating timing disruption caused by DAF. The relatively good

results obtained by some subjects who adopted such strategies provide clues in this

regard. Also experience with DAF may contribute to reduce its negative effects on

music performance, as the results of the three long-lasting experienced organists

seem to suggest. For what concerns the hypothesis that musicians would accentuate

the feedback modalities which are not altered, this is shown to be true for the tactile

feedback, whereas individual differences prevent from drawing general conclusions

about body movement.

In general, this study confirms that music playing is a matter of direct involvement

with music, in the sense of “corporeal immersion in sound energy, which is a direct

way of feeling musical reality” (Leman, 2007, p. 4). Indeed, the lack of congruency

between action and perception provoked by DAF impedes the experience of behav-

ioural resonance with the physical energies in the environment, fundamental to per-

mit the players to convey expressive intentions through music. As a consequence,

musical performance under DAF results to be impaired in all its expressive parame-

ters. On the contrary, the absence of auditory feedback does not significantly impair

performance. In this case, in fact, no misleading information is received through sen-

sory feedback, and the auditory imagery may replace the missing information basing

on previously built up statistics. Also in the absence of feedback, however, the nor-

mal behavioural resonance with the external energies is impeded: this fact results in

some negative effects of auditory feedback deprivation on the expressive nuances of

the performance (Repp, 1999).

According to the embodied music cognition, the mediator between the physical ener-

gies in the environment and the inner space of the players is body movement. There-

fore, it is not surprising that changes in the auditory feedback condition reflect on the

players’ movement. In the case of DAF, the incongruence between action and per-

ception introduced by the auditory feedback provokes embodied responses that seem

to differ according to individual variability. Further investigations should attempt to

clarify these relationships. In the case of auditory feedback deprivation, on the other

hand, the lack of sound waves causes a significant decrease in the intensity of

86

movement. Once again, this fact may be explained with the impossibility of a normal

resonance with the energies in the environment.

87

CHAPTER 7 – REFERENCE BIBLIOGRAPHY

Allen, D. R. (2007). Mental representations in clarinet performance: connections be-

tween auditory imagery and motor behaviours. Master’s thesis, University of

North Carolina, Greensboro, NC.

Banton, L. J. (1995). The role of visual and auditory feedback during the sight-

reading of music. Psychology of Music, 23, 3-16.

Barbosa, A. (2003). Displaced Soundscapes: A Survey of Network Systems for Mu-

sic and Sonic Art Creation. Leonardo Music Journal, 13, 53-59.

Bartlette, C., Headlam, D., Bocko, M., & Velikic, G. (2006). Effect of Network La-

tency on Interactive Musical Performance. Music Perception, 24, 49-62.

Bernstein, N. (1967) The Co-ordination and Regulation of Movements. New York:

Pergamon.

Black, J. W. (1951). The effect of delayed side-tone upon vocal rate and intensity.

Journal of Speech and Hearing Disorders, 16, 56-60.

Bortoli, L., & Robazza, C. (1990). Apprendimento motorio: concetti e applicazioni.

Rome: Edizioni Luigi Pozzi.

Bowmann, J. P., & Combs, C. M. (1969). Cerebellar responsiveness to stimulation of

the lingual spindle afferent fibers in the hypoglossal nerve of the rhesus mon-

key. Experimental Neurology, 23, 537-543.

Bradshaw, J. L., Nettleton, N. C., & Geffen, G. (1971). Ear differences and delayed

auditory feedback: Effects on a speech and a music task. Journal of Experimen-

tal Psychology, 91, 85-92.

88

Brunswik, E. (1956). Perception and the Representative Design of Psychological

Experiments. Berkeley: University of California Press.

Burrows, D., ed. (1997). The Cambridge Companion to Handel. New York: Cam-

bridge University Press.

Card, S., Moran, T., & Newell, A. (1983). The Psychology of Human-Computer In-

teraction. Hillsdale, NJ: Lawrence Erlbaum Associates.

Chafe, C., Gurevich, M., Leslie, G., & Tyan, S. (2004). Effect of Time Delay on En-

semble Accuracy. In Proceedings of the International Symposium on Musical

Acoustics. Nara, Japan.

Chew, E., Zimmermann, R., Sawchuk, A. A., Kyriakakis, C., Papadopoulos, C.,

François, A. R. J., Kim, G., Rizzo, A., & Volk, A. (2004). Musical Interaction

at a Distance: Distributed Immersive Performance. In Proceedings of the Mu-

sicNetwork Fourth Open Workshop on Integration of Music in Multimedia Ap-

plications, pp. 15-16. Los Angeles, CA.

Cohen, B., Goto, K., Shanzer, S., & Weiss, A. H. (1965) Eye movements induced by

electrical stimulation of the cerebellum in the alert cat. Experimental Neurol-

ogy, 13, 145-162.

Dahl, S. (2005). On the beat: Human movement and timing in the production and

perception of music. Ph.D. dissertation, KTH, Royal Institute of Technology,

Stockholm, Sweden.

Dahl, S., & Bresin, R. (2001). Is the player more influenced by the auditory than the

tactile feedback from the instrument? In Proceedings of the COST G-6 Confer-

ence on Digital Audio Effects, pp. 194-197, Limerick, Ireland.

89

Ericsson, K. A., Krampe, R. Th., & Tesch-Römer, C. (1993). The role of deliberate

practice in the acquisition of expert performance. Psychological Review, 100,

363-406.

Fairbanks, G., & Guttman, N. (1958). Effects of delayed auditory feedback upon ar-

ticulation. Journal of Speech and Hearing Research, 1, 333-346.

Finney, S. A. (1997). Auditory feedback and musical keyboard performance. Music

Perception, 15, 153-174.

Finney, S. A. (1999). Disruptive effects of delayed auditory feedback on motor se-

quencing. Doctoral dissertation, Brown University, Providence, RI.

Finney, S. A. & Palmer, C. (2003). Auditory feedback and memory for music per-

formance: Sound evidence for an encoding effect. Memory & Cognition, 31,

51-64.

Finney, S. A., & Warren, W. H. (2002). Delayed auditory feedback and rhythmic

tapping: Evidence for a critical interval shift. Perception & Psychophysics, 64,

896-908.

Föllmer, G. (2002). Musikmachen im Netz Elektronische, ästhetische und soziale

Strukturen einer partizipativen Musik. Ph.D. thesis, Martin-Luther-Universität

Halle-Wittenberg, Germany.

Freed, A., Chaudhary, A., & Davila, B. (1997). Operating Systems Latency Meas-

urement and Analysis for Sound Synthesis and Processing Applications. In

Proceedings of the International Computer Music Conference. Thessaloniki,

Hellas: ICMA.

Fuchs, A., & Kornhuber, H. (1969). Extraocular muscle afferents to the cerebellum

of the cat. Journal of Physiology, 200, 713-722.

90

Gates, A., & Bradshaw, J. L. (1974). Effects of auditory feedback on a musical per-

formance task. Perception and Psychophysics, 16, 105-109.

Gates, A., Bradshaw, J. L., & Nettleton, N. (1974). Effect of different delayed audi-

toryfeedback intervals on a music performance task. Perception and Psycho-

physics, 14, 21-25.

Gibson, J. J. (1979). The Ecological Approach to Visual Perception. Boston: Hough-

ton Mifflin.

Goebl, W., & Palmer, C. (2008). Tactile feedback and timing accuracy in piano per-

formance. Experimental Brain Research, 186, 471-479.

Hämäläinen, P., Mäki-Patola, T., Pulkki, V., & Airas, M. (2004). Musical computer

games played by singing. In G. Evangelista & I. Testa (eds), Proceedings of the

Seventh International Conference on Digital Audio Effects. Naples, Italy.

Haslinger, B., Erhard, P., Altenmuller, E., Schroeder, U., Boecker, H., and Ceballos-

Baumann, A.O. (2005). Transmodal sensorimotor networks during action ob-

servation in professional pianists. Journal of Cognitive Neuroscience, 17, 282-

293.

Havlicek, L. L. (1968). Effects of delayed auditory feedback on musical perform-

ance. Journal of Research in Music Education, 16, 308-318.

Held, R., & Hein, A. (1963). Movement-produced stimulation in the development of

visually guided behaviour. Journal of Comparative & Physiological Psychol-

ogy, 56, 872-876.

Hickok, G., Buchsbaum, B., Humphries, C., & Muftuler, T. (2003). Auditory-motor

interaction revealed by fMRI: speech, music, and working memory in area Spt.

Journal of Cognitive Neuroscience, 15, 673-682.

91

Highben, Z., & Palmer, C. (2004). Effects of auditory and motor mental practice in

memorized piano performance. Bulletin of the Council for Research in Music

Education, 159, 58–65.

Hommel, B., Müsseler, J., Aschersleben, G. & Prinz, W. (2001). The theory of event

coding (TEC): A framework for perception and action planning. Behavioral

and Brain Sciences, 24, 849-937.

Howe, M. J. A., Davidson, J. W., & Sloboda, J. A. (1998). Innate gifts and talents:

Reality or myth? Behavioural & Brain Sciences, 24, 849-937.

Howell, P. (2004). Effects of delayed auditory feedback and frequency-shifted feed-

back on speech control and some potentials for future development of pros-

thetic aids for stammering. Stammering Research, 1, 34-43.

James, W. (1890). The principles of Psychology. New York: Holt.

Janer, J. (2008). Singing-driven Interfaces for Sound Synthesizers. PhD thesis, Uni-

versitat Pompeu Fabra, Barcelona, Spain.

Jones, M. R. (1976). Time, our lost dimension: Toward a new theory of perception,

attention, and memory. Psychological Review, 83, 323-355.

Kalmus, H., Denes, F., & Fry, D. B. (1955). Effect of delayed acoustic feed-back on

some non-vocal activities. Nature, 175, 1078.

Kapur, A., Wang, G. E., Davidson, P., & Cook, P. R. (2005). Interactive Network

Performance: a dream worth dreaming? Organised Sound, 10, 209-219.

Keele, S. W. (1968). Movement control in skilled motor performance. Psychological

Bulletin, 70, 387-403.

92

Keele, S. W., & Summers, J. J. (1976). The structure of motor programs. In G. E.

Stelmach (Ed.), Motor Control: Issues and Trends (pp. 109-142). New York:

Academic.

Keller, P. E., & Janata, P. (2009). Review of “Embodied Music Cognition and Me-

diation Technology” by Marc Leman. Music Perception, 26, 289-292.

Krampe, R. T., Mayr,U., & Kliegl, R. (2005). Timing, sequencing, and executive

control in repetitive movement production. Journal of Experimental Psychol-

ogy: Human Perception and Performance, 31, 379-397.

Kristeva, R., Chakarov, V., Schulte-Mönting, J., & Spreer, J. (2003). Activation of

cortical areas in music execution and imagining: A highresolution EEG study.

NeuroImag, 20, 1872-83.

Langheim, F. J. P., Callicott, J. H., Mattey, V. S., Duyn, J. H., & Weinberger, D. R.

(2002). Cortical systems associated with covert musical rehearsal. NeuroImage,

16, 901-908.

Lashley, K. S. (1951). The problem of serial order in behaviour. In L. A. Jeffress

(Ed.), Cerebral Mechanism in Behaviour: The Hixon Symposium (pp. 112-

136). New York: Wiley.

Lee, B. S. (1950). Effects of delayed speech feedback. Journal of the Acoustical So-

ciety of America, 22, 824-826.

Leman, M. (2007). Embodied Music Cognition and Mediation Technology. Cam-

bridge, MA: MIT Press.

Lotze, M., Scheler, G., Tan, H. R. M., Braun, C., & Birbaumer, N. (2003). The musi-

cian’s brain: Functional imaging of amateurs and professionals during perform-

ance and imagery. Neuroimage, 20, 1817-29.

93

MacKay, D. G. (1987). The organization of perception and action. New York:

Springer-Verlag.

Mäki-Patola, T. (2005). Musical Effects of Latency. Suomen Musiikintutkijoiden, 9,

82-85.

Mäki-Patola, T., & Hämäläinen, P. (2004). Latency Tolerance for Gesture Controlled

Continuous Sound Instrument without Tactile Feedback. Proceedings of the In-

ternational Computer Music Conference. Miami, FL.

Mereu, S. W. (1995). Improving Depth Perception in 3D Interfaces with Sound.

Master’s thesis, University of Waterloo, Ontario, Canada.

Michels, U. (1994). Atlante di Musica. Milan: Sperling & Kupfer.

Moelants, D., Demey, M., & Leman, M. (2009). Performing music with delayed

auditory feedback. In Proceedings of the 2009 European Society for the Cogni-

tive Sciences of Music Conference (ESCOM). Jyväskylä, Finland.

Nicoletti, R. (1992). Il controllo motorio. Bologna: Il Mulino.

Palmer, C. (1997). Music Performance. Annual review of psychology, 48, 115-138.

Pfordresher, P.Q. (2003). Auditory feedback in music performance: Evidence for a

dissociation of sequencing and timing. Journal of Experimental Psychology:

Human Perception and Performance, 29, 949-964.

Pfordersher, P. Q. (2004). Can altered auditory information affect planning? Evi-

dence from music performance. Stammering Research, 1, 51-53.

Pfordresher, P. Q. (2005). Auditory feedback in music performance: The role of me-

lodic structure and musical skill. Journal of Experimental Psychology: Human

Perception and Performance, 31, 1331-1345.

94

Pfordresher, P. Q. (2006). Coordination of perception and action in music perform-

ance. Advances in Cognitive Psychology, 2, 183-198.

Pfordresher, P. Q., & Benitez, B. (2007). Temporal coordination between actions and

sound during sequence production. Human Movement Science, 26, 742-756.

Pfordresher, P. Q., & Palmer, C. (2002). Effects of delayed auditory feedback on

timing of music performance. Psychological Research, 66, 71-79.

Pfordresher, P. Q., & Palmer, C. (2006). Effects of hearing the past, present, or future

during music performance. Perception & Psychophysics, 68, 362-376.

Pöppel, E. (1997). A hierarchical model of temporal perception. Trends in Cognitive

Sciences, 1, 56-61.

Rasch, R. A. (1979). Synchronization in performed ensemble music. Acustica, 43,

121-131.

Rasch, R. A. (1988). Timing and synchronization in ensemble performance. In J.

Sloboda (Ed.), Generative Processes in Music (pp. 70-90). Oxford: Clarendon

Press.

Repp, B. H. (1999). Effects of auditory feedback deprivation on expressive piano

performance. Music Perception, 16, 409-438.

Rizzolatti, G., Fogassi, L., & Gallese, V. (2001). Neurophysiological mechanisms

underlying the understanding and imitation of action. Nature Reviews Neuro-

science, 2, 661-670.

Robinson, G. M. (1972). The delayed auditory feedback effect is a function of speech

rate. Journal of Experimental Psychology, 95, 1-5.

95

Rumelhart, D. E., & Norman, D. A. (1982). Simulating a skilled typist: A study of

skilled cognitive-motor performance. Cognitive Science, 6, 1-36.

Sawchuk, A. A., Chew, E., Zimmermann, R., Papadopoulos, C., & Kyriakakis, C.

(2003). From remote media immersion to distributed immersive performance.

Proceedings ACM SIGMM Workshop on Experiential Telepresence (ETP).

Berkeley, CA.

Schmidt, R. A., & Lee, T. D. (1999). Motor control and learning: a behavioral em-

phasis (3rd ed.). Champaign, IL: Human Kinetics.

Sheft, S. (2008). Envelope Processing and Sound-Source Perception. In W. A. Yost,

R. R. Fay, & A. N. Popper (Eds), Auditory Perception of Sound Sources (pp.

233-279). New York: Springer.

Sloboda, J. A. (1982). Music performance. In D. Deutsch (Ed.), The psychology of

music (pp. 479–496). New York: Academic Press.

Smith, A. D. (2001). WAI-KNOT (Wireless Audio Interactive Knot). Master’s the-

sis, MIT Media Laboratory, Cambridge, MA.

Thompson, W. F., Dalla Bella, S., & Keller, P. E. (2006). Music Performance. Ad-

vances in Cognitive Psychology, 2, 99-102.

Varela, F. J., Thompson, E., & Rosch, E. (1991). The Embodied Mind. Cambridge,

MA: MIT Press.

Weinberg, G. (2002). Interconnected Musical Networks: Bringing Expression and

Thoughtfulness to Collaborative Music Making. Ph.D. dissertation. MIT Media

Laboratory, Cambridge, MA.

96

97

APPENDIX 1 – INDIVIDUAL SUBJECTS’ GRAPHICS

In next pages, all the graphics concerning the experimental results of the individual

subjects are reported.

98

A1.1 Subject 2

2 - Tempo (bpm) per Measure

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay

2 - Average MIDI Key Velocity per Measure

0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay

2 - Average Head IoM (m/s^3) per Measure

00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay

2 - Average Chest IoM (m/s^3) per Measure

00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay

Figure A1.1 Subject 2: tempo, average velocity, head and chest intensity of move-

ment in the three experimental conditions.

99

Figure A1.2 Subject 2: head pitch and normalized FFT of the 4-zero-padded resam-

pled head pitch in the three experimental conditions.

100

A1.2 Subject 3


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay



101



102

A1.3 Subject 4


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay



103



104

A1.4 Subject 5

5 - Tempo (bmp) per Measure

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay



105



106

A1.5 Subject 6


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay



107

Figure A1.10 Subject 6: head pitch and normalized FFT of the 4-zero-padded re-

sampled head pitch in the three experimental conditions.

108

A1.6 Subject 7


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay



109



110

A1.7 Subject 8


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay



111



112

A1.8 Subject 9


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay



113



114

A1.9 Subject 10


20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay



115



116

A1.10 Subject 11


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay



117



118

A1.11 Subject 12

12 - Tempo (bmp) per Measure

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo Audio Delay



119



120

A1.12 Subject 13


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15




121



122

A1.13 Subject 14


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15




123



124

A1.14 Subject 15


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15




125



126

A1.15 Subject 16


0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


0102030405060708090

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

NormalNo AudioDelay


00,10,20,30,40,50,60,70,80,9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15




127



128

129

APPENDIX 2 – MEASUREMENT OF THE DELAY TIMES OF

THE CHURCH ORGAN OF ST. ANNA IN GHENT

In this section, I will present a report on an experiment that took place in St. Anna

church in Ghent, Belgium. Aim of the experiment was to measure the temporal inter-

vals that elapse between the keystrokes and the feedback perception when playing

the organ of the church (Figure A2.1).

Figure A2.1 Prospect of the St. Anna church organ.

130

The peculiarity of the St. Anna organ is the tubular-pneumatic key action (here the

term action means system of moving part; key action is the system which allows the

wind to blow into a pipe): the opening of the pipe valves is due to pressure changes

caused by the moving of small puffs of air through the tiny tubes which link the keys

to the windchests. The mechanism is well described by Figure A2.2 and Figure A2.3,

that show the effect of a keystroke on the air inside the organ. When the key is

pressed, a puff of air moves into the tube, reaching the end close to the pipes, where

the tube is mechanically connected to the back pressure channel situated below the

pipe toe-holes and the stop channels (see Figure A2.2). The back pressure channel,

full of compressed air, is separated from the stop channels and the pipe toe-holes by

a membrane which is kept pressed on by the air. The puff pressure in the tube causes

the back pressure channel to deflate, so that the membranes under the pipes deflate

too (Figure A2.3). The compressed air in the stop channels is therefore allowed to

flow into the back pressure channel, and consequently into the pipes, producing the

sound.

Figure A2.2 A membrane drawer of the organ with the key action in rest position

(adapted from Steenbrugge, 2005).

131

Figure A2.3 A membrane drawer of the organ when the key is pressed (adapted from

Steenbrugge, 2005).

The tubular-pneumatic action is a major source of delay in sound production. An im-

portant role, in particular, is played by the transmission of the key pressure through

the tubes. Steenbrugge (personal communication) estimated the time needed for the

tubular transmission in the St. Anna organ at approximately 30 ms. This estimate is

based on the fact that the tubes are approximately 10 metres long, a distance that

pressure waves cover in about 30 ms, considering a velocity of propagation in the air

at normal temperature of 340 m/s. A longer transmission time is needed for a group

of stops, such as the Principal 8’, whose pipes are positioned some metres farther

than the others from the keyboards. Other amounts of delay are due to the time

needed for the deflation of the back pressure channel and the membranes. Lastly, an

amount of delay should be proportional to the pipes dimensions.

The experiment consisted in the measurement of the time that elapses between key-

strokes and sound perception, for three different keys (a low, a middle, and a high-

frequency note) and seven different stop combinations.

132

A2.1 Method

Essentially, measurement of the delays was performed calculating the difference be-

tween the moment in which the keys were pressed and the onset of the corresponding

notes. A contact microphone was positioned over the key to be tested (see Figure

A2.4), to record the noise due to the keystrokes, whereas a Shure microphone was

positioned close to the organ pipes (see Figure A2.5), to record the produced sound.

The positioning of the Shure microphone close to the pipes, instead of close to the

console, was chosen to facilitate the signal analysis: the consequent introduction of

an additional amount of delay will be neglected in this study, due to considerations

about its order of magnitude. The two microphones were synchronized and recorded

via a Max/MSP patch.

Figure A2.4 A contact microphone is positioned over the key to be tested, to record

the noise due to the keystrokes.

133

Figure A2.5 A Shure microphone is positioned close to the pipes, to record the organ

sound.

For what concerns the keystrokes noise, I took as production moments the onsets of

the waveforms recorded by the contact microphone (Figure A2.6), which correspond

to the start of the keystrokes. Considering the fact that the pneumatic transmission

pulse starts as soon as a key is approximately 1 mm down (Steenbrugge, personal

communication), that is when the key has covered about 15% of its trajectory, taking

the onsets as the start instants introduces a certain error in the measurement. To mini-

mize such error, keypressing movements were executed quickly. On average, key-

stroke movement from the upper position to the lower one lasts around 10 ms: the re-

sulting error is so therefore approximately around 1-2 ms for each measurement. For

what concerns the sound of the organ recorded by the Shure microphone, the mo-

ments of sound production were extracted via onset detection on the corresponding

waveforms (Figure A2.7). Onset detection was performed manually with the help of

Sonic Visualiser and Audacity software.

134

Figure A2.6 Waveform of the noise generated by a keystroke. The vertical marker

indicates the onset moment.

Figure A2.7 Waveform of a Bourdon 8’ g3 recording. The vertical marker indicates

the onset moment.

Tested keys were the lowest pitch key of the keyboard, a C, the higher pitch one, a G,

and the middle C. I will refer to these keys with the notation adopted in Steenbrugge

(2005): C for the lowest C, c1 for the middle C, and g3 for the higher G. The pitch of

the corresponding notes depends on the kind of stop: for 8’ stops, C, c1 and g3 keys

correspond respectively to C2, C4 and G6 notes in the scientific pitch notation; for 4’

stops, which sound an octave above, they correspond to C3, C5 and G7. The keys no-

135

tation and their relative pitches for 8’ and 4’ stops are presented schematically in Ta-

ble A2.1 and A2.2, together with the theoretical open-pipe lengths (calculated with

the formula frequencylength 172 , where frequency refers to the fundamental fre-

quency; Steenbrugge, personal communication).

Key Pitch (scientific notation) Frequency (Hz) Theoretical pipe length (m)

C C2 65.406 2.630

c1 C4 261.63 0.657

g3 G6 1568.0 0.110

Table A2.1 Notation, pitch, fundamental frequency and theoretical open-pipe length

of the tested keys in the case of 8’ stops.

Key Pitch (scientific notation) Frequency (Hz) Theoretical pipe length (m)

C C3 130.81 1.315

c1 C5 523.25 0.329

g3 G7 3136.0 0.055

Table A2.2 Notation, pitch, fundamental frequency and theoretical open-pipe length

of the tested keys in the case of 4’ stops.

Tested stops were the single stops Montre 8’, Bourdon 8’, Gamba 8’, Prestant 4’, and

Principal 8’, plus the two stop combinations Montre 8’ plus Prestant 4’ and Montre

8’ plus Prestant 4’ plus Gamba 8’ plus Bourdon 8’. In total, the different stop combi-

nations tested were 7. For each key-stop combination, three delay measurements

were performed, giving a total of 63 measurements (3 measurements 3 keys 7

stop combinations).

A2.2 Results

Results are reported in Table A2.3 and graphically shown in Figure A2.8, Figure

A2.9, and Figure A2.10. Figure A2.8 shows average values and standard deviations

136

of the delay measurements for the 5 single stops, whereas Figures A2.9 and A2.10

show a comparison between the average values and the standard deviations for the

two stop combinations and the single stops composing them.

Stops Average Delay per key St. Deviation per key

C c1 g3 C c1 G3

Montre 8’ 0,141 0,106 0,085 0,004 0,005 0,005

Prestant 4’ 0,152 0,105 0,065 0,004 0,004 0,003

Principal 8’ 0,139 0,134 0,132 0,004 0,004 0,004

Bourdon 8’ 0,145 0,101 0,080 0,005 0,004 0,005

Gamba 8’ 0,124 0,102 0,065 0,006 0,004 0,004

Montre+Prestant 0,147 0,112 0,067 0,011 0,006 0,004

Montre+Prestant+Gamba+Bourdon 0,147 0,104 0,092 0,006 0,010 0,006

Table A2.3 Average values and standard deviations of all the delay measurements.

Figure A2.8 Mean values and standard deviations of the delay measurements for the

5 tested single stops. Each error bar is two standard deviations long.

137

Figure A2.9 Mean values and standard deviations of the delay measurements for the

Montre, Prestant, and Montre plus Prestant stops. Each error bar is two standard de-

viations long.

Figure A2.10 Mean values and standard deviations of the delay measurements for

the Montre, Prestant, Bourdon, Gamba, and Montre plus Prestant plus Bourdon plus

Gamba stops. Each error bar is two standard deviations long.

138

A2.3 Discussion

Results show that delay values vary in the range of 60-160 ms, with higher values for

the low-frequency key (C) and lower values for the high-frequency key (g3). The in-

vestigation regarding the precise nature of the dependency of the delay on the key

and the stop is beyond the scope of this experiment. Indeed, in order to clarify this

relationships, more keys should have been studied, as well as more information about

the pneumatic mechanisms and the pipes dimensions collected. However, some ob-

servations can be made. First, a dependency of the delay on the played keys is evi-

dent for all stops but the Principal 8’, for which the differences between the keys are

very small. Second, a dependency of the delay on the different stops is also clear. In

particular, considering the single stops, Principal 8’ differs from the other stops in the

high-frequency and in the middle-frequency keys, in which case it is affected by de-

lays around 135 and 130 ms, whereas the other stops show delays inferior to 115 ms

(Montre 8’ and Bourdon 8’) or 90 ms (Prestant 4’ and Gamba 8’). Gamba 8’ presents

a shorter delay than the other stops in the low-frequency key, while for the high-

frequency key Gamba 8’ and Prestant 4’ both present values lower than the others.

Montre 8’ and Bourdon 8’ are the only two stops that show a very similar behaviour

over all the three keys. For what concerns the stop combinations, somehow unex-

pected is the fact that the two combinations show opposite behaviours: delay values

for the Montre plus Prestant combination are similar to the lower between the values

of the single stops composing it, whereas delay values for the Montre plus Prestant

plus Bourdon plus Gamba are similar to the higher. The third observation is that a

constant amount of delay seems to be present over all the measurements, arguably

due to the transmission time.

Again, an investigation of the reasons underlying the dependency of the delay on

keys and stops is beyond the scope of the present study. In general, a kind of propor-

tionality between the delay and the diameter of the pipes (see Figure A2.11) should

be expected. This relation, anyway, should be not straightforward, as the measure-

ments seem to confirm. In particular, a striking difference has been registered be-

tween the delay values of the Montre 8’ and the Principal 8’, which have almost

identical pipe dimensions. A partial explanation for this difference lies in the fact that

the Principal pipes are positioned farther than the other pipes, therefore they need a

139

longer transmission time. Nevertheless, this fact does not account for the apparently

very different asymptotical behaviours.

Figure A2.11 External diameters of the pipes for Montre 8’, Prestant 4’, Principal 8’

and Gamba 8’ (values from Steenbrugge, 2005).

A2.4 Conclusions

The experiment shows that the amounts of delays in the St. Anna pneumatic organ

are prominently high. Referring to the discussion in Subparagraph 1.3.2, all meas-

ured delays are beyond the break-point interval, individuated as the delay length for

which auditory feedback starts to be perceived as delayed and DAF disruption be-

comes significant. In particular, delay values for the low-frequency notes are very

close to the 170-200 ms window which was found to be very damaging in many

studies on music performance (Havlicek, 1968; Bradshaw, Nettleton & Geffen, 1971;

Gates & Bradshaw, 1974; Pfordresher & Benitez, 2007; Moelants, Demey & Leman,

2009), and often considered as the critical interval in speech (Black, 1951; Fairbanks,

1955; Butler & Galloway, 1957; Fairbanks & Guttman, 1958; MacKay, 1968; How-

ell, Powell, & Khan, 1983; Fabbro & Darò, 1995). Moreover, when playing the or-

gan, two more factors contribute to make performance very difficult. First, attack

140

times have also to be taken into account: for low-frequency notes, attack phase can

last more than 100 ms (e.g., for the Bourdon 8’ stop), so that the overall time needed

for a clear pitch recognition may reach 250 ms or more. Second, the strong rever-

beration of the sound waves in the church further complicate the relationships be-

tween played notes and their perception. Playing St. Anna’s organ therefore consti-

tutes a very challenging task, at least for players who don’t have experience under

analogous conditions, and pneumatic organs such as St. Anna’s one can be taken into

consideration as appropriate instruments for ecological experiments on music per-

formance under DAF.

A2.5 Reference bibliography

Bello, J. L., Daudet, L., Abdallah, S. A., Duxbury, C., Davies, M. E., & Sandler, M.

B. (2005). A Tutorial on Onset Detection in Music Signals. IEEE Transactions

on Speech and Audio Processing, 13, 1035-1047.

Black, J. W. (1951). The effect of delayed side-tone upon vocal rate and intensity.

Journal of Speech and Hearing Disorders, 16, 56-60.

Bradshaw, J. L., Nettleton, N. C., & Geffen, G. (1971). Ear differences and delayed

auditory feedback: Effects on a speech and a music task. Journal of Experimen-

tal Psychology, 91, 85-92.

Butler, R. A., & Galloway, F. T. (1957). Factoral analysis of the delayed speech

feedback phenomenon. Journal of the Acoustical Society of America, 29, 632-

635.

Fabbro, F., & Darò, V. (1995). Delayed auditory feedback in polyglot simultaneous

interpreters. Brain and Language, 48, 309-319.

Fairbanks, G. (1955). Selective vocal effects of delayed auditory feedback. Journal

of Speech and Hearing Disorders, 20, 333-346.

141

Fairbanks, G., & Guttman, N. (1958). Effects of delayed auditory feedback upon ar-

ticulation. Journal of Speech and Hearing Research, 1, 333-346.

Gates, A., & Bradshaw, J. L. (1974). Effects of auditory feedback on a musical per-

formance task. Perception and Psychophysics, 16, 105-109.

Havlicek, L. L. (1968). Effects of delayed auditory feedback on musical perform-

ance. Journal of Research in Music Education, 16, 308-318.

Howell, P., Powell, D. J., & Khan, I. (1983). Amplitude contour of the delayed signal

and interference in delayed auditory feedback tasks. Journal of Experimental

Psychology: Human Perception and Performance, 9, 772-784.

MacKay, D. G. (1968). Metamorphosis of a critical interval: Age-linked changes in

the delay in auditory feedback that produces maximal disruption of speech.

Journal of the Acoustical Society of America, 43, 811-821.

Michels, U. (1994). Atlante di Musica. Milan: Sperling & Kupfer.

Moelants, D., Demey, M., & Leman, M. (2009). Performing music with delayed

auditory feedback. In Proceedings of the 2009 European Society for the Cogni-

tive Sciences of Music Conference (ESCOM). Jyväskylä, Finland.

Pfordresher, P. Q., & Benitez, B. (2007). Temporal coordination between actions and

sound during sequence production. Human Movement Science, 26, 742-756.

Shannon, J. R. (2009). Understanding the Pipe Organ: A guide for Students, Teach-

ers, and Lovers of the Instrument. Jefferson, NC: McFarland & Company.

Steenbrugge, D. (2005). Het Pierre Schyven orgel in de Gentse Sint-Annakerk. Or-

gelkunst, 111.

142

Documents

Matteazzi Master's Thesis 2009