Teleclassing

Opzet & Technologische uitdagingen

Bart De Schuymer

Agenda

• Project goal

• Technological challenges

Agenda

• Project goal

• Technological challenges

Project goal

Bologna declaration

29 European countries

reform higher education

bachelor/master system

but also: joined forces between universities

solve student’s mobility problem

increasing interest in teleclassing:

synchronous communication between classrooms

4

Project goal

IBBT-Teleclassing: distance education in Flanders

joined forces of VUB and UGent

in cooperation with regional industry

small classrooms

combination of two teleclasses

ex cathedra

teacher-to-student/student-to-teacher interaction

5

Project goal

immersive

cfr. telepresence:

but in classrooms:

6

Project goal

original goals

user’s research

related, specific technological developments

fully functional, realistic setup

7

Project goal

budget shortages due to

economic crisis

change of government/university policy

restriction to lab demonstrator

realistic setup

real lessons with teachers & students

user’s research

8

Project goal

9

local site

Project goal

10

remote site

Agenda

• Project goal

• Technological challenges

Setup

12

Network

13

Network Requirements to support a

Teleclassing environment

Teleclassing – characteristics :

High Quality / High Definition

Real-Time

Interactive

Network Requirements to support a

Teleclassing environment

Quality of Service definitions :

Smartboard

16

Interactive whiteboard

Central application for teaching a class

Based on Smartboard technology

One whiteboard per class room

Synchronised

Functionality

Support of multiple types of media

Annotations

Annotations on top of media

“classical” whiteboard function

Collaboration between whiteboards is possible

Teacher keeps overall control

18

Media

Powerpoint

PDF

Images

Movies

3D content

19

Example

20

Audio/Video encoding & streaming

21

Streaming components

Using open standards

Hardware interfacing (Camera’s and microphones)

Capture cards

Encoders

DirectShow based software encoders

Video: H.264/AVC (SD)

Audio: AAC

Streaming protocol: RTP/RTCP

One RTP/RTCP stream per media stream

Decoders

Regular media players

Streaming server

Darwin Streaming Server

22

Capture

Standard pan-tilt-zoom cameras

23

Visualisation

Pixel-canvas

Standard projectors

24

Results lab demonstrator

Encoders streaming to streaming server

Streams available for multiple clients

Round trip delay: approximately 600ms

Capture cards

Inherent delay encoding

Buffering decoders

25

Advanced HD video coding using H.264/AVC FRExt.

Immersive teleclassing experience requires

transmission of multiple high resolution video streams

between locations

This consumes a significant amount of bandwidth

Efficient compression is needed

Improve compression performance of H.264/AVC

coding for HD material

Adaptive quantization

Post-filtering for quantization noise suppression

26

Adaptive quantization

Contrast sensitivity of the human eye:

More sensitive to low spatial

frequencies than to high spatial

frequencies

More sensitive to luma (intensity) than

to chroma (color) information

Coefficients representing high spatial

frequencies and color information are

represented with less accuracy

Accuracy/quantization determined based

on model of the contrast sensitivity of the

HVS.

Up to 5% bit-rate savings for the same

perceptual quality compared to uniform

quantization

27

No. of

perc

eiv

able

levels

Spatial frequency (cycles/degree)

Red-Green

chrominance

Blue-Yellow

chrominance

Luminance

Post-filtering for quantization noise suppression

Lossy compression

introduces (quantization)

noise into the video frames.

Design a filter to suppress

this noise at the encoder and

send coefficients to the

decoder for post-processing

Studied state-of-the-art filter:

up to 12% bit-rate reduction

for the same quality can be

obtained.

28

+Filtering+

+-

s t

n t

x t

x t h t

s t

s t

e t

h(t) chosen such that e2 is minimized

bitstream

bitstream

Encoder Decoder

Filter calculation

&encoding

Filter reconstruction

&post-filtering

Ori

gin

al

De

co

de

d

Filt

ere

d

De

co

de

d

Audio

29

Audio capture

teacher

standard wireless headset

remote students

microphone array

developed during the project

30

Audio Captation for Teleclassing

31

Internet

Requirements for audio captation in a classroom setup

Good SNR

Physically robust

Flexible

Immersive

User-friendly

Simple maintenance

Single Microphone vs. Microphone Array

32

0.2

0.4

0.6

0.8

1

30

210

60

240

90

270

120

300

150

330

180 0

0.5

0.75

1

30

210

60

240

90

270

120

300

150

330

180 0

Omnidirectional microphone

Directive microphone

Flexible

No Noise

suppression

Good Noise

Suppression

Fixed speaker

position

Microphone array

0.25

0.25

0.5

0.75

1

30

210

60

240

90

270

120

300

150

330

0

Flexible (steerable)

Good Noise suppression

Application 1: Noise Suppression

Multi-channel Noise Suppression

Microphone Array Beamforming

e.g.: 8-microphone linear array

Environmental noise: up to 18dB SNR gain

Localised noise: up to 25dB SNR gain

Single-Channel Noise Suppression

Spectral Subtraction

Interference Cancellation

33

0 0.2 0.4 0.6 0.8 1

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

original speech

time (s)0 0.2 0.4 0.6 0.8 1

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

single microphone

time (s)0 0.2 0.4 0.6 0.8 1

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

microphone array

time (s)

MA+ Spectral Subtraction

0 0.2 0.4 0.6 0.8 1

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

time (s)

34

Application 2: Acoustic Echo Suppression

Internet

Multi-channel Echo Suppression

Microphone Array Beamforming

Single-channel Echo Suppression

Interference Cancellation

0 0.5 1 1.5 2 2.5

x 105

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

single microphone

time (s)0 0.5 1 1.5 2 2.5

x 105

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

microphone array

time (s)0 0.5 1 1.5 2 2.5

x 105

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

MA + interference canceller

time (s)

Student

Teacher

Application 3: Reverberation Suppression

Multi-Channel Reverberation Suppression

Microphone Array Beamforming

Microphone Arrays: Small to Big

36

Microphone Arrays come in many different shapes and sizes:

From 2 to 1020 microphones

From 6 cm to 3 m wide

Linear, circular, arc shaped,...

Microphone array: realisation

technical requirements

small size

non-intrusive

can be integrated

changeable listening direction

good audio quality

patent application filed

37

Microphone array: realisation

lab demonstrator setup

38

Microphone array: realisation

later prototypes

39

Microphone array: realisation

directivity

40