Interfaces Montréal

Audio and video IP

Jeremy Cooperstock

McGill University, Centre for intelligent machine (cim)

From Videoconferencing to shared reality

Anyone who has used videoconferencing tools, ranging from simple desktop applications such as CU-SeeMe or Netmeeting, to high fidelity professional systems, quickly realizes that videoconferencing is not the same as physical presence, nor even to a telephone call. While the conversants can see video images of each other, these are often of limited quality. Worse, the latency in the audio signal results in an unnatural “turn-taking” style of conversation that diminishes the quality of interaction and exagerates the sense of distance. This is all the more serious for tasks that demand synchrony, for example, distributed music performance applications.

In order to overcome these problems, we are developing a new research facility known as the Shared Reality Environment. The primary goal of this environment is to support the exchange of very low-delay and high-fidelity audio and video streams between multiple users in different locations, in order to engender users with a rich sense of co-presence. Part of this development effort has led to the release of our Ultra-Videoconferencing system, which provides a flexible, low-latency IP transport for audio, video, and most recently, vibrosensory data. We have used this system for a range of demanding applications including live concert streaming (1999), remote mixing (2000), collaborative performance (2001), distance masters classes over SDI (2002), and remote video interpreting of sign language using three simultaneous DV streams (2003). The presentation will demonstrate several aspects of this research, applications to distributed collaboration, as well as the associated research challenges.

Jeremy Cooperstock is an associate professor in the department of Electrical and Computer Engineering, a member of the Centre for Intelligent Machines , and a founding member of the Centre for Interdisciplinary Research in Music Media and Technology at McGill University. His research interests focus on computer mediation to facilitate high-fidelity human communication and the underlying technologies that support this goal. Cooperstock directs the Shared Reality Lab, a space that provides distributed individuals the experience of being in the same room at the same time, and leads the technical development of the Ultra-Videoconferencing system. His accomplishments include the Intelligent Classroom, the world’s first Internet streaming demonstrations of Dolby Digital 5.1, uncompressed 12-channel 96kHz/24bit, multichannel DSD audio, and low-latency videoconferencing, as used by maestro Pinchas Zuckerman for distance violin teaching and by musicians at two ends of the continent for a distributed jazz performance.

Cross-continental jazz session: Musicians at McGill play with Stanford University (right, on screen) with end-to-end delay of less than 50ms.