ANET Workshop

     History of Internet Audio Experiments at McGill (and elsewhere)

Jeremy R. CooperstockCentre for Intelligent Machines

Centre for Interdisciplinary Research in Music Media and Technology

McGill University

ANET Workshop

Software available from:

http://ultravideo.mcgill.edu

Research supported by:

ANET Workshop

Some Early (Pre-)History

Eve Schooler: Distributed Music: A Foray into Net Performance (Sept. 1993)

synchronized three real-time streams from different hosts; delays in the order of 200 ms difficult for performers to be listeners

Paul Hoffert: CyberSoiree (Feb. 1996) ATM-based technology for audio and video streaming of a four-way jazz performance delay >0.5s delay but musicians earned to compensate through extensive practice

Dimitri Konstantas: Distributed Musical Rehearsal Studio (May 1996) ATM based distributed rehearsal with conductor at different location from musicians 80 ms one-way delay for audio-video synch; echo resulted in "extreme confusion"

Seiji Ozawa: Opening Ceremony Nagano Winter Olympics (1998) conduct choruses on 5 continents: singers in Sydney, New York, Beijing, Berlin, False Bay time lag adjustor used to eliminate satellite delay

ANET Workshop

World's first Internet AC-3 streamMontreal-New York for the AES 107th Convention, Sept. 26, 1999

ANET Workshop

System Diagram

1.5

Mbps

1.5 Mbps

SGI Indy

PC ClientCisco IP/TVPC Server

SGI IndyNetwork

Dolby encoder

AC-3 (640 Kbps) (5.1 channels,16 bits @ 48kHz)

coded in AES/EBU

McGill UniversityMontreal, Canada

NYU Cantor HallNew York, USA

ANET Workshop

In action

Receiver

Transmit Receive,check, playback

Send AudioQueue

Sender

Receive AudioQueue

Read

ANET Workshop

Recording Studio that Spanned a ContinentMontreal-Los Angeles for the AES 109th Convention, Sept. 23, 2000

McGill Redpath Hall, Montreal

USC Norris Theatre, LA

12 channels, 96kHz, 24

bits/channel

ANET Workshop

Microphone Configuration

ANET Workshop

3 Mbps

ANET Workshop

27.6 Mbps

ANET Workshop

Some Modern History

Internet2 Initiative: World's First Remote Barbershop Quartet (Nov. 2000) multi-location quartet; each of the 4 singers in different cities, conductor in 5th network delay variances prevented singers from hearing each other or conductor

Internet2 Initiative: Music Video Recording via Internet2 (Nov. 2000) multi-location music video recording session using real-time streaming video musicians simultaneously connected via timing tracks to a mixing board

Chris Chafe: QoS Enabled Audio Teleportation (Nov. 2000) CD quality sound (750 kbps) of 2 separated musicians in Dallas streamed to Stanford musicians played "together" in same space (Stanford) but delay was severe

John Wawrzynek: Network Musical Performance (May 2001) gestural coding (e.g. MIDI) used to manage data for distributed musical performance musicians at Berkeley and CalTech, playing on MIDI keyboards; local feedback only

ANET Workshop

"Frères Jacques" experimentMontreal, RISQ 2000 Conference

"low-latency" MPEG-2 videoconferencing

ANET Workshop

"Playing Together" experimentNew York-Ottawa, Remote Masters class, Dec. 8, 2000

used Litton MPEG-2 codec @ 10 Mbps

Pinchas Zuckerman, Columbia U, NYC

Wu Ji, Canarie ARDNOC, Ottawa

ANET Workshop

Quality vs. Latency

near-equivalent quality of encoded video at greatly reduced bandwidth

but... compression takes time

0

50

100

150

200

250

300

uncompressedSDI

M-JPEG/DV MPEG-4

Bandwidth (Mbps)

ANET Workshop

Tolerated Synchronicity

compress transmit

100 20 4030 50 7060 80ms

speed of light: Mtl-LA

net latency Mtl-LA

networks

smallensemble

large ensemble

lip synch

musicalactivity

* This figure is a theoretical “best-case” scenario. Actual performance may vary based on local hardware. The manufacturer makes no warranties, either express or implied, and will not be held liable for damage resulting from use of the product. Certain restrictions may apply. Offer void where prohibited by law. Batteries not included.

MPEG*acquire

Latency:Latency: The Interaction-KillerThe Interaction-Killer

or put another way...

ANET Workshop

Interactive Success!

Chris Chafe: SoundWIRE (August 2001) streaming pro audio from remote locations for musical collaboration UDP/IP streaming: good results between Armonk, NY and Stanford round trip delay 125ms, musicians able to "catch-up" during pauses

ANET Workshop

The future according to Sympatico?

ANET Workshop

For those with DSL at home...For those with DSL at home...

yeah, right!(tu n'es pas

sérieux!)

ANET Workshop

So what does it take?

much bigger network pipes ("abusive bandwidth") need Fast Ether (100Mbps) for audio GigE (1000Mbps) for video more efficient network topologies – hops add delay

ANET Workshop

So what does it take?

much bigger network pipes ("abusive bandwidth") need Fast Ether (100Mbps) for audio GigE (1000Mbps) for video more efficient network topologies – hops add delay

more powerful hardware CPU speed (fast process switching, especially for mixed media) pro-audio interfaces with small buffers

ANET Workshop

So what does it take?

much bigger network pipes ("abusive bandwidth") need Fast Ether (100Mbps) for audio GigE (1000Mbps) for video more efficient network topologies – hops add delay

more powerful hardware CPU speed (fast process switching, especially for mixed media) pro-audio interfaces with small buffers

more masochistic researchers

ANET Workshop

Sometimes things work...

ANET Workshop

Sometimes things work...

and sometimes they don't

note: look of extreme anguish

ANET Workshop

Distributed Violin DuetMcGill-Université de Montréal for RISQnet Conference, Nov. 21, 2001

connected two Montreal universities via IP over 1 Gbps fiber link

uncompressed audio and video

< 30ms latency

ANET Workshop

"The medium is the message" Marshall McLuhan

Cooperstock's Corollary:

"The medium must be faster than the message."

ANET Workshop

Size does matterMontreal-Ottawa (NRC), Distance Teaching Trial, Feb. 20, 2002

connected McGill to National Research Council (Ottawa) with SDI video and multichannel 96kHz/24bit audio

display on 50" plasma screen (near life-size)

ANET Workshop

Distributed Jazz JamMontreal-Stanford (CCRMA), Research Demo, Oct. 24, 2002

uncompressed audio and video @ < 50ms latency

ANET Workshop

Remote Video Interpretation (Sign Language)Montreal-Fredericton (UNB), research trials, 2003

DV (patient, physician) exchanged with remote sign-language interpreter

physician's setup

patient's setup

ANET Workshop

Vibrosensory TransmissionVRQ demo, March 8, 2004

ANET Workshop

Coming next: DSD TransmissionAES demo, October 31, 2004

ANET Workshop

Milestones Ahead

multicast traffic shaping region-of-interest client selection multichannel echo-supression/cancellation

Self-Signal Isolation: Echo-suppression

the big problem in videoconferencing same issue for video and vibrosensory data

achieving high-fidelity distributed interaction, at levels of presence and latency that support the

most demanding applications and doing so in spite of sensor and bandwidth limitations

D:\video\sre\SRE.mp4

http://ultravideo.mcgill.edu