Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music

Conductors

W. David Hairston, Ph.D

Advanced Neuroscience Imaging Research LabDepartment of Radiology; Wake Forest University School

of Medicine, Winston-Salem, NC 27157

Multisensory enhancement within a number of paradigms, including:

• Simple reaction times

• Saccadic response latencies

• Signal detection

• Orientation/localization behavior in cat

General conditions for enhancements to be observed

-spatial alignment

-temporal congruence

-minimal efficacy (“inverse effectiveness”)

Stimuli:

Vis (LED)

Aud (broadband)

Vis-Aud

50 ms, 30 reps

What about human localization ability

Hairston et al, J. Cog. Neuro, 2003

0

10

20

30

40

50

60

70

80

-15 -13 -11 -9 -7 -5 -3 -1 1 3 5 7 9 11 13 15

Error (degrees)

Pe

rce

nt

Visual

Multisensory

Auditory

0

5

10

15

20

25

30

35

-15 -10 -5 0 5 10 15

Error (degrees)

Pe

rce

nt

Evaluating Precision

Precision ~ Std Dev

0

2

4

6

8

10

12

-40 -20 -10 0 10 20 40

Target Position (deg.)

Lo

cali

zati

on

Var

iab

ilit

y (d

eg.)

Visual

Auditory

Multisensory

Hmmmm…. Why?

A

C

B

S1 S2 M

The total amount of gain/ enhancement observed is determined by the relative contribution of each sense, based upon its own perceptual acuity

How do you increase auditory acuity?

Conductors as auditory localizers

Daily, career experience requires rapid and accurate assessment of auditory scene

Also requires coordination of multisensory information – “who” played “what” wrong note, etc

20 conductors, min 7 years “podium experience” (ave 10.2)

Matched on age, education, sex, etc

(Hodges, Hairston & Burdette ’06)442.8

454.2

440

445

450

455

460

465

470

440 Hz Base Tone

Fre

quency

(H

z)Conductors

Non-Conductors

A#

A

0

1

2

3

4

5

6

7

8

0 10 20 30 40

Target Location (+/- deg.)

Pre

cisi

on

(+

/- d

eg.)

Auditory

Multisensory

Visual

Visual and multisensory performance very similar

Non-musicians

Multisensory performance enhanced

0

1

2

3

4

5

6

7

8

0 10 20 30 40

Target Location (+/- deg.)

Pre

cisi

on

(+

/- d

eg.)

Auditory

Multisensory

Visual

Control

0

1

2

3

4

5

6

7

8

0 10 20 30 40

Target Location (+/- deg.)

Pre

cisi

on

(+

/- d

eg.)

Auditory

ControlImproved auditory performance

0

2

4

6

8

10

12

-7 -2 3 8 13

% Auditory Improvement (> Control)

% M

ulti

sen

sory

E

nh

ance

men

t (>

Vis

ual

)

10

40

3020

Conclusions from this…

• While untrained subjects gain little from an additional auditory cue, music conductors appear to benefit from additional auditory signals, for which their spatial acuity is much better.

• The degree of multisensory gain in this unique population appears to be tied to their improved auditory performance

• These results suggest that the specialized training and experience of these individuals has a profound affect within both the auditory and multisensory realms

But what if…

• Multisensory “enhancements” are beneficial when the task can make use of additional information

• BUT - When do not match or are not relevant – can be detrimental or inhibiting….

-Slower RT

-Biased localization

-Illusions and misperceptions

Many times the task at hand required focusing on one sense and ignoring other.

“Tuning out” irrelevant information…

Current studies…

• Physiological effects of “tuning in” to one sense over another

• “Cross-modal deactivation”: Decreased activity within one sensory cortex related to stimulation of another

E.g., decreased BOLD signal in occipital cortex during auditory task

• Some evidence for relation to selective attention

Unclear whether the extent of deactivation observed is related to the difficulty of the task at hand

MethodsGoals:

Does changing task difficulty affect cross-modal deactivation?

Does extreme acuity within one modality, and unique multisensory training affect this process?

Subjects- Non-musicians – various careers, no formal musical training, 25-40 y/o- Conductors – min. 5 yrs podium experience

Matched on age, gender, education, etcWhy conductors?

- Heightened auditory acuity - Conducting activities require NOT inhibiting other

(e.g., visual) information

Methods: Tasks

Pitch Discrimination

440 Hz 660 Hz

440 Hz 660 Hz

Easier

Difficult

60 ms

20 ms

Temporal Discrimination

440 Hz 456 Hz

440 Hz 443 Hz

Easier

Difficult(Time)

Base Test

Subjects’ thresholds for each task acquired prior to fMRI scanning

Methods: Thresholding

Acquired threshold

0

20

40

60

80

100

120

1 6 11 16 21 26 31 36 41 46 51

Presentation orderS

OA

(m

s)

Allows compensation for variability in perceptual acuity between subjects and groups

2 down/1 up rule

Perceptual acuity, non-musicians vs. conductors

430

440

450

460

470

Conductors Non-musicians

Fre

qu

ency

(H

z)

0

20

40

60

80

100

Conductors Non-musicians

SO

A (

ms)

A

A#

Base comparison

Conductors

Non-Musicians

Non-musicians need a significantly larger difference to discriminate tones than conductors.

Methods: fMRI

Each task performed at 2 levels

- at threshold (“difficult”)- above threshold (“moderate”)

Also performed visual temporal discrimination (2 circles) Visual

Temporal discrimination

silence TRTR(Scanner ON) (Scanner ON)(Scanner OFF)

Trial Trial Trial

Sparse sampling (10 s pause), block design

Schmuel et al, Nature Neuroscience, 2006

Positive BOLD

Positive BOLD

Analysis

• Activity in task (ON) blocks contrasted against resting baseline (OFF)

Baseline: eyes open on fixation, no task• ROI of visual-responsive occipital cortex

2.78 7.0

-2.78 -7.0 Used to generate summary stats

Non-musicians: Moderate

Deactivation of visual cortex

+/-3.95 +/-8.0

Vis ROI

Non-musicians: Difficult

Robust deactivation of visual cortex

+/-3.95 +/-8.0

Vis ROI

Non-musicians: Difficult vs. Moderate

EasierDifficult

-3

-2

-1

0Voxel (Y coord)

Me

an

Sig

na

l

-3

-2

-1

0Voxel (Y coord)

Me

an

Sig

na

l

-1

-0.5

0

0.5

1

Mea

n S

igna

l (%

)

-650

-450

-250

-50

150

Tota

l Sig

nal M

agni

tude

0

200

400

600

800

1000

# of

Vox

els

Non-musicians: stats

ROI A.Cort.Post. Cing.

V Cort.

ROI

A.Cort.

Post. Cing. V Cort.

# significant voxels

Total Signal Magnitude

Mean signal

Conductors

DifficultModerate

Vis ROIVis ROI

+/-3.95 +/-8.0

Only slight deactivation of visual cortex in both cases

Conductors: Difficult vs. Moderate

Easier Difficult

0

200

400

600

800

1000

# o

f V

ox

els

-650

-450

-250

-50

150

To

tal S

ign

al M

ag

nit

ud

e

ROI A.Cort.Post. Cing.

V Cort.

# significant voxels

Total Signal Magnitude

Non-musicians vs. Conductors

Difference seen when task is difficult

Non-Musicians Conductors

0

40

80

120

160

200

Easier Difficult

N S

ign

if V

oxe

ls

Non-musicians

Conductors

n.s.

**

Difficult Task

0

20

40

60

80

High M oderate

Acc

ura

cy(%

)

Conclusions

• The degree and extent of cross-modal deactivation observed specifically depend of the task difficulty; when the same task is easier, cross-modal deactivations are attenuated.

• Contrary to non-musicians, conductors show only minor cross-modal deactivation, even when the task is very difficult to perform.

• This suggests that the role of functional deactivations is dynamic, and adapts to fit the needs of the individual and situation at hand.

Conductors

• Conductors are highly-trained individuals with unique daily activties and specialization

• This experience leads to not only high auditory acuity, but altered interactions when dealing with multiple senses

• Two theories:

- High auditory acuity negates the need for visual suppression.

- Daily experience has led to familiarity with completing complex auditory tasks while also monitoring visual information.

0

15

30

45

60

75

90

Conductors Non-musicians

0

20

40

60

80

High M oderate

Acc

ura

cy(%

)