A Kuriakidou, E Filtzantzidou, P Balaouras,I Roussakis, C. Mouzakis, I Stavrakakis (2001)

A video gateway between the next generation broadband Internet and

the current narrow band Internet for live and on-demand access

IST – 1999 – 10160

Title Academic Distance Learning Trial

Version 1.2

Deliverable # 15.4

Author A Kuriakidou, E Filtzantzidou, P Balaouras,I

Roussakis, C. Mouzakis, I Stavrakakis

Company UoA

Date 30/01/2002

Filename UoAwp6-del15.4-v1.2.doc)

Security* Int#

* Int = Internal circulation within project (+ EC PO if requested)

Rest = Restricted circulation list (specify in footnote) and EC PO only

IST = Circulation within IST Programme participants

FP5 = Circulation within Framework Programme participants

Pub = Public document

# FOOTNOTE: space for circulation list if required (delete this otherwise)

TITLE Academic Distance Learning Trial

VERSION 2

DELIVERABLE # 15.4

AUTHOR A Kuriakidou, E Filtzantzidou, P Balaouras,I Roussakis, C. Mouzakis, I Stavrakakis

COMPANY UoA

DATE 30/01/200

page 2 of 49

TABLE OF CONTENTS

INTRODUCTION 3

Description of the application 3

ROADMAP TO TRIALS 4

Trial design and pedagogical evaluation methology 4

Testing equipment 4

Setting the laboratory environment for Scenarios II-IV. 4

DESCRIPTION OF TRIALS 5

SCENARIO I 5

TRIAL IA 6

Technical Configuration 9

TRIAL IB 10


TRIAL IC 12


ScENARIOS II, III 15

Results of Scenario II & III 17

SCENARIO IV: PERFORMANCE EVALUATION 17

Performance of the MGW-2000 System 17

Performance of the MPEG-4 Transmission Server and ISO MPEG-4 Player 19

Results of Scenario IV 20

TRIALS THEORY AND RESULTS 21

A. – Learning and Instruction – Some Preliminary Remarks 21

B. – Pedagogical Evaluation of Distance Education – Forms and Criteria Applied in the Trials 22

C. – Evaluation Results 25

C.1 – Evaluation of Pedagogy / Instruction 25

C.2 – Pedagogical / Perceptual Evaluation of Technical Features 27

C.3 – Scenic Direction 31

C.4 - A statistical analysis of the factors affecting the satisfaction of the participants 32

GENERAL SUGGESTIONS – CONCLUDING REMARKS 33

APPENDIX QUESTIONNAIRE 35


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 3 of 49

INTRODUCTION

This deliverable is the evaluation report of the UoA trials.

Description of the application

UoA have conducted a series of distance learning trials in which students participated as well

as testing trials executed in the laboratory. These trials have implemented four (4) different

scenarios. The objective of the distance learning trials was to evaluate VideoGateway’s ability

to provide live lecture material to geographically disparate users at rates ranging from as low as

500 kbps for desktop users up to 4 Mbps for classroom users. The testing trials aimed to:

(a) evaluate the VideoGateway system performance bounds, the MPEG-4 transmitter

and the associated players

(b) verify the transcoding and rate adaptation features which were part of the R&D

effort of the project.

Let us introduce the scenarios:

The first scenario – scenario I – was focused on the distance learning trials in which more than

120 undergraduate and graduate students, research and technical staff participated. This

scenario utilised the encoding features – PAL to MPEG-1, MPEG-2 and MPEG-4 streams of

the VideoGateway project products.

The trials have engaged campus based students who attended live academic lectures of both the

undergraduate and postgraduate programmes of the Department of Information and

Telecommunications. Technical staff of the UoA’s Network Operation Centre participated in

these trials as well. The population of users for each trial test is illustrated in the tables that

accompany each trial scenario.

The MGW-2000 and MPEG-4 transmitters were used to deriver the various streams in terms of

encoding formats and rates. Since it was hard to simultaneously broadcast a lecture and co-

ordinate the large number of involved users/students, it was decided to carry out three (3) trials

focused on scenario I, instead of one (1) as originally planned, to better manage the trials

themselves.

The evaluation of all the trials of scenario I was based on user input (perceptual) through

appropriate forms.

The second scenario – scenario II – was combined with the third scenario – scenario III – and

aimed to evaluate the transcoding and rate adaptation features of the Transcoding technology

developed in the VideoGateway project. This scenario was implemented in a laboratory

environment.

The fourth scenario – scenario IV – was carried out in the laboratory environment aiming to

stretch and investigate the system performance limitations of the encoding features

Deliverable 15.5 provides a common summary of all the VideoGateway project trials.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 4 of 49

ROADMAP TO TRIALS

The actions taken to prepare the trials are briefly reported below:

TRIAL DESIGN AND PEDAGOGICAL EVALUATION METHOLOGY

Contact Prof. D. Mathaiou and his research team – Department of Primary Education,

UoA – for preparing the pedagogical evaluation methology and the questionnaire.

Fine tuning of the trial settings.

Determination of the required equipment and ordering it from Optibase.

Contact and invite partners from GUnet to participate in the trial (Athens University

of Economics and Business, National Technical University of Athens, Aristotle

University of Thessaloniki).

Redesign the trials according to the project progress, results and products.

Select lectures for delivery.

Select users.

TESTING EQUIPMENT

Communication with Optibase for equipment shipment.

Communication with Greek Customs to get the equipment.

Using, managing and testing the equipment.

Integrating the equipment with the UoA teleteaching classroom infrastructure.

SETTING THE LABORATORY ENVIRONMENT FOR SCENARIOS II-IV.

Install a linux based routing system with 3 NICs.

Install and test for enabling multicast routing.

Install the NIST emulator.

Test the rate adaptation algorithm before the integration with Optibase’s transcoder.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 5 of 49

DESCRIPTION OF TRIALS

In this section the trials are organised in Scenarios (I to IV). The Scenarios described are the

following:

I) Encoding Features

II) Transcoding Features

III) Rate Adaptation

IV) System Stretching.

For each Scenario, a number of different trials and tests were executed. The Scenarios (trials

and tests) are described in terms of equipment used, network, stream parameters, etc.

SCENARIO I

As mentioned in the Introduction, the first scenario – scenario I – was focused on the distance

learning trials. Instead of simultaneously broadcasting a lecture at different rates and due to the

complication of co-ordinating the large number of involved users/students, it was decided to

carry out three different (3) trials focused on scenario I (Figure 1). Let Ia, Ib and Ic refer to

these trials.

Scenario I – Scenario I – utilising encoding featuresutilising encoding features

UoA classr oom

VGV ideo

Matrix

PAL

M PEG -1, MP EG-4

to Desktop users at 500 kbpsP ict ur eT e l

P ict ur eT e l

P ict ur eT el

MPE G-1, MPEG-4

to Desktop users at 1 Mbps

P ic tu re T el

P ic tu re T e l

P ic tu re T e l

MP EG -2, MPE G-1, MPEG-4

to Classroom

AUEB classr oom

Ia

Ib

Ic

Figure 1 Trials Ia, Ib and Ic of Scenario I

The aim of Scenario I was to investigate the potential of exploiting products that deliver

MPEG-2, MPEG-1 and MPEG-4 streams for the deployment of distance learning services. The


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 6 of 49

targeted users are students located at remote classrooms and students at remote PC laboratories

(desktop users). Thus, the technical part of scenario I utilised the encoding features – PAL to

MPEG-1, MPEG-2 and MPEG-4 streams of the Video Gateway project products. Let us

examine each distance learning trial separately:

Trial Ia was focused on a lecture delivered to a remote classroom. The video and

audio streams of the instructor were transmitted as MPEG-2, MPEG-1 or H.323

compliant streams. The presentation slides were in electronic form and separately

transmitted to the remote classroom via the network as well. An H.323 audio-visual

channel was used as return channel for permitting interaction between the two

classrooms.

Trial Ib was focused on a lecture delivered to a remote classroom, as well. The video

and audio streams of the instructor were transmitted as MPEG-1, and MPEG-4. The

presentation slides were in electronic form and separately transmitted to the remote

classroom via the network as well. No return channel for permitting interaction

between the two classrooms was available. Trial Ib, in comparison to trial Ia, engaged

in addition the MPEG-4 format. The MPEG-1 stream in trial Ib was transmitted at a

lower rate than the rate of the MPEG-1 transmission in trial Ib.

Trial Ic was focused on a lecture delivered to desktop users. The video and audio

streams of the instructor were transmitted as MPEG-1 and MPEG-4 but in lower rates

than the trial Ib. The presentation slides were printed-out and provided to the students

before the beginning of the transmission. The presentation slides were not transmitted

separately in electronic form but they were part of the video view that was transmitted

No return channel was available.

The trials have engaged campus based students who attended live academic lectures of both the

undergraduate and postgraduate programmes of the Department of Information and

Telecommunications. Technical staff of the UoA Network Operation Centre participated in

these trials as well. The population of users for each trial test is illustrated in Table 1.

The evaluation of all the trials of scenario I was based on user input (perceptual) through

appropriate forms. The detailed description of each trial of scenario I is presented below.

Table 1: Participants

Trial Number of participants

Ia 42

Ib 60

Ic 22

Total 124

TRIAL IA

Trial Ia was held on 12th July 2001. Dr. Xandrinos started his lecture at 1:30 pm and lasted for

1 hour and 30 minutes. The lecture took place in the UoA teleteaching room and was

transmitted to a similar room in the Athens University of Economics and Business (AUBE). 18


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 7 of 49

students were located in the instructor’s room and 24 students in the remote room. The lecture

was divided in three sections, each of duration of 30 minutes:

Section A. The instructor’s video and audio were transmitted as multiplexed MPEG-2

stream at a rate of 4 Mbps. The presentation slides were shared via the MS

NetMeeting – a T.120 Application Sharing software.

Section B. The instructor’s video and audio were transmitted as multiplexed MPEG-1

stream at a rate of 1.5 Mbps.

Section C. The instructor’s video and audio were transmitted by utilising H.323

audio-visual systems at a rate of 1.5 Mbps.

The software and equipment used during Trial Ia was the following.

Equipment:

Optibase – MGW-2000, VideoPlex MPEG-1 and MPEG-2 hardware decoding card.

Other – VCON’s H.323 audio-visual terminal

Software:

Optibase – MGW-2000 Director, ComMotion Receiver 4.0

Other – MeetingPoint VCON, MS NetMeeting

In more detail, the MGW-2000 was used to transmit the MPEG-1 and MPEG-2 streams from

the UoA classroom to the AUEB classroom. The VideoPlex hardware decoding card and the

ComMotion Receiver 4.0 player were used to receive the aforementioned streams and decode

them into analogue form. VCON’s H.323 audio-visual terminal was used during the entire

session as a return channel to enable the communication between the two classrooms and as the

main channel as well during section C. The software of MS-NetMeeting (T.120 application

sharing session) was used in order for the instructor to share his presentation slides with the

remote classroom. For the application sharing session two Personal Computers were used.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 8 of 49

TrialTrial Ia Ia : 12/07/2001 Dr. : 12/07/2001 Dr. XandrinosXandrinos

UoA classr oom

V ideo

Matrix

PA L

M PEG -1 and M PEG -4

to Desktop usersP ic tu reT e l

P ic tur eT e l

P ic tur eT e l

MPE G-1, MPEG-4 stream

to Desktop users

P ic tu re T el

P ic tu re T e l

P ic tu re T e l

MPEG -2@ 4M bps ,

MPEG -1@ 1.5 Mbps

H.263 @ 1.5 Mbps to Classroom

AUEB classr oom

24 S tudents

MGW2000

Figure 2: Trial Ia

155 A TM

UoA

C isco L S 1010

C isco 7513

H ercu le s

F ore A S X-1000

T ele -ed uca tion class room

@ Dept. o f In for matics

T e le -ed ucation classroom

@ AU E B

AU EB

GRNet

F ore A S X-1000

12 M bps

A UE B

m ain rou ter

8 M bps

Figure 3: Network Infrastructure


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 9 of 49

Technical Configuration

Section A.

The instructor’s video and audio were transmitted in MPEG-2 format. The MGW 2000 Director

configuration was set as follows:

Source

Video Input Composite

Hue

Saturation 128

Brightness 128

Contrast 128

Horizontal Offset 0

Video Horizontal Filter 6

Video Vertical Filter 7

Audio Input Unbalanced

Audio Gain 3

Stream

Colour System PAL

Encoding Mode Full D1

Stream Format MPEG Transport

Send System Header 0

Target Bit Rate 4 Mbps

Video Frame Sampling Full

Audio Encoding Mode Stereo

Audio Bit Rate 192 Kbps

Audio Sampling Rate 44100 Hz

Target

Current Target 1

Target Type UDP

Target Address Remote room’s address (unicast)

Port 11111

UDP Block Size 1024

TTL 10

The students’ video and audio were transmitted with the H.323 terminal at 384 Kbps (video)

and 128 kbps (audio). The instructor’s material was sent separately with a T.120 application.

(NetMeeting).

Section B

The instructor’s video and audio were transmitted in MPEG-1 format. The MGW-2000

Director configuration was set as before with the only difference in the Stream format – MPEG

System.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 10 of 49

Section C

The instructors’ video and audio were transmitted with the H.323 terminal at 1.5 Mbps for the

video stream and 128 kbps for the audio.

TRIAL IB

Trial Ib was held on 11th October 2001. Prof. Stavrakakis started his lecture at 10:30 pm and its

duration was 1 hour. The lecture took place in the UoA teleteaching room and was transmitted

– over a loaded network – to another UoA classroom equipped with audio-visual equipment. 10

students were located in the instructor’s room and more than 50 students in the remote room.

The lecture was divided in two sections of duration of 30 minutes:

Section A. The instructor’s video and audio were transmitted as split MPEG-4

streams at a rate of 1 Mbps. The presentation slides were shared via the MS

NetMeeting, a T.120 Application Sharing software.


stream at a rate of 1 Mbps. The presentation slides were shared via the MS

NetMeeting, a T.120 Application Sharing software.

TrialTrial Ib Ib : 11/10/01: 11/10/01 Prof Prof. I. . I. StavrakakisStavrakakis

UoA classr oom

V ideo

Matrix

PA L

MPE G-1, MPEG-4

to Desktop users

P ic tu re T el

P ic tu re T e l

P ic tu re T e l

MPEG -1@ 1 Mbps

MPEG -4 @ 1 Mbps to C lassroom

UoA classroom

50 S tudentsMG W2000

MPEG -4 Transmission

Server

Figure 4: Trial Ib

The software and equipment used during Trial Ib was the following.

Equipment:

Optibase – MGW-2000, MPEG-4 Transmission Server.

Other – Audio-visual equipment


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 11 of 49

Software:

Optibase – MGW-2000 Director, ComMotion Receiver 4.0, ISO-MPEG-4 player

Other – MS NetMeeting

In more detail, the MGW-2000 was used to transmit the MPEG-1 stream from UoA’s

teleteaching classroom to the other classroom. The ComMotion Receiver 4.0 player was used to

receive the MPEG-1 stream and decode it. The stream was rendered in its digital form and not

in analogue form as in Trial Ia. The MPEG-4 Transmission Server was used to transmit the

MPEG-4 stream from the UoA teleteaching classroom to the other classroom. The ISO-MPEG-

4 player was used to receive the MPEG-4 stream and decode it. The stream was rendered in its

digital form. The software of MS-NetMeeting (T.120 application sharing session) was used in

order for the instructor to share his presentation slides with the remote classroom. For the

application sharing session two Personal Computers were used.


Section A.


Director configuration was set as follows.

Source


Hue

Saturation 128

Brightness 128

Contrast 128

Horizontal Offset 0




Audio Gain 3

Stream

Color System PAL

Encoding Mode SIF

Stream Format MPEG System


Target Bit Rate 1 Mbps





Target

Current Target 1

Target Type Multicast

Target Address 224.1.1.1

Port 11111

UDP Block Size 1024


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 12 of 49

TTL 10

Section B

The instructor’s video and audio were transmitted in MPEG-4 format. The configuration was

set as follows:

Description Teleteaching

RTP Address 224.1.1.77

Port 11200

Preview File Video Optibase4.mp4

Preview Track 1

Video Live Encoding Parameters:

Video Capture Device Video Device 1

Bitrate (bps) 1000000

Intra Period 30

Picture Description SIF

Frame per Second 25

Audio Live Encoding Parameters:

Audio Capture Device Audio Device 1

Bitrate (bps) 64000

Sample per Second 44100

Channel Mode Stereo

TRIAL IC

Trial Ic was held on the 2nd November 2001. A pre-recorded lecture (the lecture of Prof.

Stavrakakis in trial Ib) was transmitted in two labs of the Department of Informatics and

Telecommunications and to the Network Operation Centre. The transmission started at 15:00

and lasted about 1 hour. There were 20 participants, postgraduate students and research and

technical staff. The lecture was divided into two sections of duration 30 minutes:

Section A. The instructor’s video and audio were transmitted as split MPEG-4

streams at a rate of 500 kbps. The participants received the lecture with the ISO

MPEG-4 Player.


stream at a rate of 500 kbps. The participants received the lecture with the

ComMotion Receiver.

The presentation slides were not shared via the MS NetMeeting as in the previous trials but

instead the users downloaded and printed the slides before the transmission.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 13 of 49

Trial Trial IcIc : 2/11/01 : 2/11/01 ProfProf . I. . I. StavrakakisStavrakakis

UoA classr oom

V ideo

Matrix

PA L

20 P artic ip ants

MG W2000

MPEG -4 Transmission

Server

MP EG -1, MPE G-4

to Desktop users at 500 kbps

P ict ur eT el

P ict ur eT el

P ict ur eT el

Figure 5: Trial Ic

The software and equipment used during Trial Ic was the following.

Equipment:

Optibase – MGW-2000, MPEG-4 Transmission Server.

Other – Audio-visual equipment

Software:

Optibase – MGW-2000 Director, ComMotion Receiver 4.0, ISO-MPEG-4 player

In more detail, the MGW-2000 was used to transmit the MPEG-1 stream from the UoA

teleteaching classroom to the desktop PCs. The ComMotion Receiver 4.0 player was used to

receive the MPEG-1 stream and decode it. The MPEG-4 Transmission Server was used to

transmit the MPEG-4 stream from the UoA teleteaching classroom to the other classroom. The

ISO-MPEG-4 player was used to receive the MPEG-4 stream and decode it.


Section A.

The instructor’s video and audio were transmitted in MPEG-4 format. The configuration was

set as follows:

Description Teleteaching

RTP Address 224.1.1.77

Port 11200

Preview File Video Optibase4.mp4

Preview Track 1

Video Live Encoding Parameters:


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 14 of 49

Video Capture Device Video Device 1


Intra Period 30

Picture Description SIF

Frame per Second 25

Audio Live Encoding Parameters:

Audio Capture Device Audio Device 1


Sample per Second 44100

Channel Mode Stereo

Section B


Director configuration was set as follows.

Source


Hue

Saturation 128

Brightness 128

Contrast 128

Horizontal Offset 0




Audio Gain 3

Stream

Colour System PAL

Encoding Mode SIF



Target Bit Rate 500 kbps





Target

Current Target 1

Target Type Multicast

Target Address 224.1.1.1

Port 11111

UDP Block Size 1024

TTL 10


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 15 of 49

SCENARIOS II, III

The scenario II, as defined in WP2, aimed to utilise the transcoding features of the Video

Gateway, that were the “on the fly” transcoding of MPEG-1 streams to MPEG-4 streams.

During the project it became clear to the partners that the stream transcoding feature would not

give any additional advantages from the user/student point of view in comparison to the trials of

Scenario I, where MPEG-4 was evaluated. Thus it was decided that the scenario II to

demonstrate the transcoding technology developed in the context of the VideoGateway project

by the real time transcoding of MPEG-1 files to MPEG-4 stream.

Pictu reTel

1. Uo A Dum p clie nt,

2 . Bit R ate Verifie r

M P EG -1 file

M P EG -1 to M PE G-4

rate adap table t ranscoder

PictureT el

M P EG -4

pla yer

linux system

NIS T em ulator

m routed (for mu lticast)

IP m ulticast add res s: 2 24.1.1.77

V id eo p ort: 1 1200

O D port: 11130

Dem o targets:

- T ranscoding MPE G-1 to MP EG-4

- Transcoding & R ate Adaptation o f MP EG-4 output stream s

PC1

PC2

PC3

PC4

Figure 6: Transcoding and rate adaptation trial

Scenario III aimed to demonstrate the rate adaptation features of the MPEG-4 sources by

incorporating the R&D results of Optibase and UoA, concerning the rate adaptation algorithms

both at the encoding/transcoding and the network side.

Scenarios II and III were combined and a single trial was conducted using an MPEG-1 to

MPEG-4 transcoder capable of producing rate adaptable MPEG-4 video streams.

The architecture of the trial is that presented in Figure 6.

Optibase has provided the MPEG-1 to MPEG-4 transcoder - based on a transcoding engine

which is capable of transcoding MPEG-1 files to an MPEG-4 split audio and video stream. The

transcoder broadcasts the two streams over multicast IP. The streams are transported via

RTP/UDP/IP packets.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 16 of 49

The ISO MPEG-4 player listens to a specific multicast IP address and port, receives the MPEG-

4 RTP/UDP/IP packets and decodes the video stream, which is displayed on the monitor. This

player does not have the capability of reporting the packet losses due to network congestion by

sending back Real Time Control Packets (RTCP) to the transcoder.

That is why UoA provided the UoA-dump client which is capable of receiving the video stream

(but not decoding) and reporting back to the transcoder the network status by RTCP Receiver

Reports (RR). The transcoder receives the RTCP RR from the UoA-dump client and adapts its

transmission rate based on the rate adaptation algorithm provided by UoA. This algorithm

implements a multi-state congestion control based on the fractional packet loss reported by

RTCP RR and the history. A detailed description of the multi-state congestion control

algorithm may be found in a paper which is part. The trials have been carried-out in live

academic lectures from both the undergraduate and the postgraduate programmes of the

Department of Information and Telecommunications to campus based students. Technical staff

of the Network Operation Centre of UoA also participated in these trials. The population of

users for its trial test is illustrated in Deliverable 9 (R&D results).

The adaptation of the transmission rate is verified by using the Bit Rate Verifier, which reports

the mean rate of the stream every 3 seconds.

During the trial the transcoding and rate adaptation features were tested in the UoA laboratory.

The transcoder, ISO MPEG-4 player, UoA-dump and Bit Rate Verifier client were installed in

three Personal Computers (PC1, PC2, PC3) running Win2k OS, that were connected to a linux

router system equipped with three NICs. The network emulator software NIST was installed.

This software was used to introduce packet losses that triggered the rate adaptation.

Pictu reTel

1. Uo A Dum p clie nt,

2 . Bit R ate Verifie r

M P EG -1 file

M P EG -1 to M PE G-4

rate adap table t ranscoder

PictureT el

M P EG -4

pla yer

linux system

NIS T em ulator

m routed (for mu lticast)PC1

PC2

PC3

PC4

RTP

RTCP R R

packet

losses

Figure 7: RTP/RTCP Packets


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 17 of 49

Results of Scenario II & III

The conclusions drawn from the execution of scenarios II & III were that both the transcoding

and rate adaptation features of the Transcoder are working properly. Another preliminary

conclusion is that indeed the MPEG-4 encoding scheme tolerates a greater fractional packet

loss in comparison to the MPEG-1 encoding scheme.

SCENARIO IV: PERFORMANCE EVALUATION

The objective of scenario IV was the stretching and performance evaluation of the MGW-2000

system, MPEG-4 Transmission Server and the ISO-MPEG-4 player components. Since, the

MPEG-1 player (ComMotion Receiver) is a well established product in the market, no

evaluation was carried out for it. It was important to stretch and evaluate the three

aforementioned systems because they may constitute basic components to an architecture that

would deploy live distance learning scenarios.

Performance of the MGW-2000 System

IP

Pic tureTel

P ictureTe l

V ideo

M atrix

PALPictureT el

MGW2000

Figure 8: Stretching of MWG-2000

Test 1: MPEG-1 transmission over a 10 Mbps port connection

The MGW 2000 was set to transmit three live MPEG-1 streams. Only one NIC was used for the

transmission and it had access to a 10 Mbps switch. The following table shows the achieved

frame rates for different values of the target bit rate.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 18 of 49

TARGET BIT RATE FRAME RATE DELAY (IN SECONDS)

3 sources at 2 Mbps 24.7-24.4 1


3 sources at 3.5 Mbps 14.5-15.5 2

1 source at 4 Mbps 24.6 1

2 sources at 4 Mbps 22.5 1.5

3 sources at 4 Mbps 10-12 4

When all sources were transmitted at 4 Mbps the system failed to respond after five minutes of

use and had to be reset. The problem was due to the fact that the aggregate output traffic rate of

the MGW-2000 was greater than that enabled by the port of 10 Mbps.

Test 2: MPEG-1 transmission over a 100 Mbps port connection

After the first test we used a 100 Mbps switch and repeated the same tests. The table below

shows these results.

TARGET BIT RATE FRAME RATE DELAY (IN SECONDS)



3 sources at 3.5 Mbps 24.4-24 1

3 sources at 4 Mbps 24.7-24 1

Thus when a proper network connection (100 Mbps) was used, no problem appeared in the

performance of the MGW-2000 system.

The MGW-2000 configuration of tests 1 and 2 is described in the tables below.

Source


Hue

Saturation 128

Brightness 128

Contrast 128

Horizontal Offset 0




Audio Gain 3

Stream

Colour System PAL

Encoding Mode SIF



Target Bit Rate 2, 3, 3.5 and 4 Mbps






VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 19 of 49

Test 3: MPEG-2 transmission over 10 and 100 Mbps port connections

The MGW-2000 was set to transmit three live MPEG-2 streams. At first the NIC had access to

a 10 Mbps switch and the results were similar with the first test (MPEG-1 transmission). When

the MGW-2000 had access to a 100 Mbps switch we were able to transmit up to 5 Mbps

successfully.

Performance of the MPEG-4 Transmission Server and ISO MPEG-4 Player

Due to its nature this trial was performed in the laboratory. Figure 9 illustrates the trial

architecture. The major components used were:

The MPEG-4 transmitter server run on a Personal Computer equipped with dual

processors at 800 MHz, 128 MB RAM and the OSPREY 200 video grabbing card.

The Win2k server OS was installed.

The receiver was a PC with a single Pentium III at 400 Mhz, 128MB memory and the

Windows Professional 2000 operating system. The ISO-MPEG-4 player was installed

in the receiver PC. The duration of each test was fifteen minutes.

The Transmitter server and receiver were connected over a 10 Mbps connection.

IP

PictureT el

PAL

MPEG-4 Transmission Server

ISO MPEG-4 Player

Receiver

Figure 9: Stretching of MPEG-4 Transmission Server and MPEG-4 ISO Player

The performance evaluation test aimed to:

evaluate the perceived quality and delay of the MPEG-4 video at different bit rates at

the receiver

record resource usage such as CPU time and memory usage at both the transmitter

(MPEG-4 encoding process) and the receiver (MPEG-4 decoding process).

It was noted that the decoding process proceeded without any problem for the bit rates of 0.5,

1.0, 1.5 and 2.0 Mbps. Beyond the bit rate of 2 Mbps the receiver (MPEG-4 player) processing

power was not adequate to successfully decode the MPEG-4 stream (see also Table 3).

The perceived video quality at the receiver was excellent. The overall delay due to the

encoding, transmission, decoding and rendering processes when movement occurred was about


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 20 of 49

1.8 seconds for bit rates of 0.5, 1.0 and 1.5 Mbps. At 2.0 Mbps the delay was about 2.1 seconds.

At higher bit rates the MPEG-4 decoder and encoder were not capable of performing the

equivalent processes. Table 3 presents the measurements for the perceived quality, the delay of

the MPEG-4 video and resource usage at the receiver. The perceptual quality is measured with

a scale from 1 to 5 ( Table 2).

Table 2:Quality measures

Excellent Quality 5

Very Good Quality 4

Good Quality 3

Bad Quality 1,2

Table 3: Results at the Receiver

BIT RATE (MBPS) 0.5 1.0 1.5 2.0

Perc. Quality 5 5 5 5

Delay (second) 1.8 1.8 1.8 2.1

CPU Time (%) 65 75-83 84-89 95-99

Memory Usage (K) 43.448K 41.9K 38.748K 44.1K

At the receiver the CPU time was increasing gradually with the bit rate. At 2.0 Mbps the

decoding process was using 99% of the CPU time. The receiver was not able to handle bit rates

greater than 2.0 Mbps. The memory usage varied between 38.7K and 44.1K.

At the transmitter the above parameters hardly changed with the bit rate increase. The CPU

time of the decoding process varied between 35 and 37 percent and only at the bit rate of 2.0

Mbps was there a slight increase and the CPU time was between 39 and 43 percent. The

memory usage parameter varied between the 19.4K and 19.6K. Table 4 presents the CPU time

and memory usage at the transmitter side for bit rates of 0.5, 1.0, 1.5 and 2.0 Mbps.

Table 4: CPU time and Memory Usage of Transmitter

BIT RATE (MBPS) 0.5 1 1.5 2

CPU Time (%) 35-37 34-38 35-37 39-43

Memory Usage (K) 19.4K 19.4K 19.5K 19.6K

Results of Scenario IV

The conclusion of this trial is that the three major components, the MGW-2000 system, the

MPEG-4 Transmission Server and the ISO MPEG-4 player do not present any performance

limitations that could prohibit the deployment of live distance learning scenarios.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 21 of 49

TRIALS THEORY AND RESULTS

A. – LEARNING AND INSTRUCTION – SOME PRELIMINARY REMARKS

When it comes to instruction we must bear in mind that:

Teaching and learning are different processes and the former does not necessarily

imply the latter.

There is a vast range of factors involved in both processes, such as the different

individual characteristics of the participants, the dynamics of the peer groups, diverse

attitudes and approaches towards teaching and learning, but also the nature of

different subject contents to be taught and learned. This renders it impossible to come

up with an “ideal” instructional approach or a “universally applicable” instructional

methodology.

The type of learning that is demanded from different disciplines / subjects also varies.

As Entwistle and Ramsden1 noted: “In arts, students should be encouraged to search

for personal meaning, which seems to depend on empathy and openness from staff,

informal teaching (discussion) methods, freedom for students to explore their

interests, and yet, because of that freedom, the setting of clear goals and standards. In

science and social science, good teaching depends more on pitching the information at

the right level and being alert to student difficulties. A deep approach in science

depends more on operational learning, on relating evidence and conclusion, and on

the appropriate use of certain amounts of initial rote learning to master the

terminology. But this versatility in learning will emerge readily only when the

workload is reasonable, and where freedom in learning is allowed. The forms of

assessment, the types of questions, will also need to be consistent with lecturers'

attempts to develop critical thinking”.

In order to achieve learning, any instructional setting, regardless its being a traditional

classroom lecture or a computer-mediated videoconference, must respond to the

learners’ needs. Otherwise the “transactional distance”2 among the key players of the

instructional process (i.e. the teacher, the learner and the knowledge / subject content)

can result in cognitive misunderstandings / misconceptions, communication gaps or

psychological pitfalls (loss of interest, withdrawal, isolation etc). In distance learning,

physical separation of teacher and learner can reinforce these kinds of problems or

can delay the proper responsive actions.

1 Entwistle, N.J. and Ramsden, P. (1983) Understanding Student Learning. Croom Helm

2 Moore, M. G. (1980) Independent Study. In Boyd R D and Apps, J. W. and Associates, Redefining the

Discipline of Adult Education. Jossey-Bass


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 22 of 49

B. – PEDAGOGICAL EVALUATION OF DISTANCE EDUCATION – FORMS AND CRITERIA APPLIED IN THE TRIALS

These unique features of learning and instruction render educational evaluation a very difficult

and subtle task. Educational researchers have identified different models of educational

evaluation. Following Alexander and Hedberg3, educational evaluation approaches fall into

four main paradigms4:

“Objective-based: Evaluation as a process of determining the degree to which educational

objectives are being achieved. This follows the scientific tradition and is straightforward to

apply, but does not take account of unintended outcomes, and takes no account of students as

individuals with all their differences.

Decision-based: Focuses on the decisions made during development and improvements that

could be made. It is useful for programmes with a large scope or multiple levels, but needs the

co-operation of decision makers. It has proved difficult to put into practice and expensive to

maintain.

Value-based: Evaluation is not only concerned with goals, but also whether the goals are worth

achieving. Formative and summative evaluation is used, and the evaluator considers major

effects, achievements and consequences of the programme. This acknowledges the importance

of unintended outcomes, and learners' perceptions of the learning experience, and evaluation

can be made without the need to know about the objectives. Its perceived disadvantages are that

it may leave important questions unanswered.

Naturalistic approach: organises evaluations around the participants' key concerns and issues.

Uses qualitative data collection such as journals, observations and interview. The advantages

are that it acknowledges context and can be used to benefit those being studied, but participants

may identify criteria with little educational worth.”

In the approach we adapted for the streaming video trials, we have tried to evaluate (a) the

parameters of the technical setting / context that we thought had a significant impact on the

educational interaction and, (b) the instruction and the learning process itself.

We conducted the evaluation using:

Questionnaires for the students who attended streaming video classes

Semi-structured interviews with the instructors

Observation grills filled during the trials by the evaluation team.

3 Alexander, A. and Hedberg, J. G. (1994) Evaluating technology-based learning: Which model? Interactive

Multimedia in University Education, In Beattie, K., McNaught, C. and Wills, S. (eds) Designing for Change in

Teaching and Learning, Proceeding of the IFIP Working Conference, Melbourne, 6-8 July 1994, Elsevier

4 cf : (1999) Desktop Videoconferencing for Tutorial Support. PhD Thesis University College London


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 23 of 49

The questions we sought concerning the evaluation of the technical context refer mainly to the

factors that influenced the attention of the learners, their willingness to sit the streaming video

courses, their perceptions about the process. The parameters we investigated were:

Parameters concerning the quality of the transmitted picture:

the size of the projected image in the remote classrooms and the desktop

computers

the size of the projection of the instructional aids (i.e. PowerPoint

presentations)

the fidelity of the projected picture (clearness, depth, colours etc.)

the transmission of the movement of the educator or the participants

(interrupted, awkward, natural etc.)

the visualisation of the communicative expressions of the participants (i.e. the

expressions on the face of the instructor) and the synchronisation of sound and

picture

the quality of the transmitted instructional material (colours, fonts, sound,

movement etc.)

Parameters concerning the quality of the transmitted sound:

the fidelity of the sound

the intensity of the sound

the hue / tone of the sound

the noise due to networking problems

the echo effect

the interruptions or the delays in the sound received in the classrooms /

desktops

Parameters concerning the effectiveness of the direction / mastering choices

the use of different camera views of the teacher or the host classroom audience

(when there is one)

the wideness of the camera view

the focus, the zoom and the interchange of views during the transmission

the use of a variety of instructional techniques and instructional materials

the movement of the instructor

Parameters concerning the classroom set up

organisational features of the host classroom (aesthetic approach, setup of the

projection stand, etc.)

functionality of used lights, microphone locations etc.

parameters concerning the technological / network infrastructure

usability / adequacy of the technological and communicational media used by


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 24 of 49

the learners

readiness to incorporate different instructional aids into the transmitted

instruction

reliability of the used electronic devices (PCs, projectors, storage systems,

control equipment etc.)

reliability of the network communication (interruptions, delays, other problems

concerning the normal transmission of sound and picture)

The questions we seek concerning the instructional process are similar to those used in the

evaluation of conventional instruction. Emphasis is on the active learning involvement of the

students and the reinforcement of their motivation, since these two elements need to be clearly

addressed in a distance learning environment5. The parameters used to evaluate the

instructional aspect of the trials were:

Parameters concerning instructional activities:

clear statement of the pursued learning objectives

organisation of the instructional materials

consideration of the students' experience

adequate organisation and use of instructional time

clear use of learning strategies

Parameters concerning communication / interaction:

communication style of the instructor

non verbal communication

use of questions

Parameters concerning the instructional material / aids

genres of instructional material / aids used

amount of information included

attractiveness

stimulation of deep learning

stimulation of higher cognitive skills

usability / adequacy

Parameters concerning the learners

individual features of the learners

interest towards taught subject

5 see ADEC, 1999) [ADEC Guiding Principles for Distance Learning and Teaching, American Distance

Education Consortium


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 25 of 49

familiarity with new technologies / streaming video

Parameters concerning the instructor

appearance of the instructor

communicative skills

instructional skills

use of technical media

C. – EVALUATION RESULTS

As mentioned above the evaluation plan involved (a) the use of questionnaires, which were

filled by the participants including both closed and open-ended questions, (b) the use of

observation grilles, which were filled by the research and evaluation team during the trials, and

(c) interviews with the tutors about their experience from the lectures. In this section we are

going to present the results from the analysis of the questionnaires and the observation grills.

The evaluation team has gathered a total of 105 questionnaires that can be used for our

purposes. This sample can not be used to draw generalisations about the pedagogical /

instructional effectiveness of the medium, but can give us a good idea of the features that affect

the process. About 80% of the respondents were male (20% female), and 74% were between

19-23 years of age (19% were from 24-28 and 7% were older than 29 years). The majority for

trials Ia and Ib were undergraduate and students (67% for trial Ia, 100% for trial Ib), while in

trial Ic the majority were graduate students (50%) and technical / academic staff (28%). All

respondents shared familiarity with popular computer programmes (such as MS Widows, MS

Office) and the Internet (all knew how to use a web browser and an e-mail client).

Again, in the first two trials, the majority of the respondents had little or no experience with

streaming video (56% for trial Ia and 72% for trial Ib), while for the third trial (Ic) 67%

admitted they were familiar with streaming video. This feature clearly identifies two different

groups of respondents (a "novice" group and an "expert" group).

The evaluation team analysed the frequencies of the responses to the questionnaires and used

the observation grills to clarify certain features of the trials that were not obvious from the

statistical analysis.

C.1 – Evaluation of Pedagogy / Instruction

Most people in the trial groups were moderately or very familiar with the thematic of the

delivered lecture (75% in Ia, 54% in Ib, 67% in Ic). But most of them were not familiar with the

tutor (this had been the first chance to attend a lecture by the tutor for 60% of the participants).

To evaluate the pedagogical / instructional adequacy of the delivered lecture, the participants

were asked to answer 17 questions about different features of the lesson. Since the views

expressed in all three cases (Ia, Ib and Ic as mentioned above) are similar, we are going to

discuss them all together, distinguishing between the different trials only when necessary.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 26 of 49

In each case the tutor was reported by the respondents to have clear-cut educational objectives

(Ia: 100%, Ib: 75%, Ic: 84.3%). The participants stated that the tutor took under consideration

the background knowledge of the students (Ia: 74.4%, Ib: 67.5%, Ic: 58.8%), delivered the

lecture in a structured manner (Ia: 100%, Ib: 73.1%, Ic: 79%) and established rapport with the

students (Ia: 70%, Ib: 56.6%, Ic: 56.3% -note that the during the first trial the tutor could

interact with the remote students). The differences are attributed to the instructional

characteristics of each tutor and the nature of the content of the lecture. These facts can lead us

to assume that for the purposes of the present evaluation all three trials can be considered as

pedagogically successful.

According to the responses of the participants:

In each trial the educational material (mainly presentation slides) that was used facilitated the

development of the topic (Ia: 92.1%, Ib: 63.4%, Ic: 45%) The differences are attributed to the

design of the material, which in some instances did not show well over the network. In most

cases "non-verbal" communication from the tutor, such as his movement or his "body language"

was helpful for the participants (Ia: 81%, Ib: 48%, Ic: 70%). The only occasion where

participants expressed their reservations about the use of instructional material or about the

movements / expressions of the tutor were when these features were disturbed due to technical

reasons (mostly scenario Ib) or when the camera could not follow these movements (mostly

scenario Ia and Ic). This is also the case with gestures and facial expressions, since in some

occasions it was difficult to understand their meaning due to the transmitted view angle

(scenarios Ia and Ic).

The duration of each trial (about 60 min overall, featuring a small intermediary break to change

stream settings) was considered by most of the participants as adequate for the delivery of a

lecture through streaming video (Ia: 58.2%, Ib: 65.8%, Ic: 72.2%). But most participants (over

60%) agree that the information / lecture content that is delivered through streaming video must

be moderately less than the amount presented in conventional classes, more carefully structured

and adequately supported by custom-made instructional aids (such as slide presentations).

As expected, the lack of the physical presence of the tutor affected the attendance of remote

students (Ia: 20%, Ib: 41.4%, Ic: 38.1%), and deteriorated the rapport that could be established

between the two parts (Ia: 43.8%, Ib: 56.1% Ic: 65%). Please note that the interactive

communication that was established during trial Ia deteriorated the negative feelings of the

participants. What is surprising, however, is that a large part of the participants (about 40%)

feel that they were marginally influenced by these factors while only about 10% were very

sceptical about them. The evaluation team attributes this feature (a) to the successful

instructional experience, as it was described above and, (b) to the experience of the students

with television, where we feel familiar with individuals who we have never met in person. In all

three trials more than 70% of the students felt that the tutor addressed equally all students

regardless their location (host or remote classroom), so that they did not feel neglected or

deprived.

In the open-ended questions, the students called for some form of interactivity between them

and the tutor especially when the lecture is demanding and there are many questions to be

addressed (mostly in trial Ib). They also called for more applied knowledge -not only

theoretical considerations- (Ib) and they emphasised the ability of the tutor to deliver the lecture

in structured, comprehensive manner (Ia and Ib).


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 27 of 49

We must note that while in trial Ia the host and the remote classroom were able to interact, the

questions which the evaluation team used to assess the pedagogical effectiveness were not

markedly difference between this trial and the other two. But from this particular trial the

participants did not call in their open-question responses for interactivity or probable

unanswered questions. On the contrary these students enjoyed their chance to participate

actively during the lecture (80%) and to collaborate with peers located remotely (32%).

C.2 – Pedagogical / Perceptual Evaluation of Technical Features

1st Trial – Scenario Ia

1st Part – MPEG-2 (4 Mbps)

The participants’ views about:

POSITIVE (%) NEUTRAL (%) NEGATIVE (%)

The size of the screen 73.7 10.5 15.8

The quality / fidelity of the picture

65.8 18.4 15.7

The quality / fidelity of the sound

62.9 25.7 21.4

Picture Interruptions: 16.7 30.6 52.7

Transmission of movements of the tutor

63.6 24.2 12.1

Echo 50 32.4 17.6

Sound Interruptions 28.2 25 46.9

Network noise 21.3 42.4 36.4

Lip synchronisation 51.4 28.1 12.5

2nd Part - MPEG 1 (1,5 Mbps)





64.7 23.5 11.8


56.3 34.4 9.4



50 37.5 12.5

Acoustics 35.5 51.6 12.9

Sound Interruptions 25.8 41.9 32.3



3rd Part – H.323 (1,5 Mbps)



The size of the screen 58.2 25.8 16

The quality / fidelity of the 49.8 27.3 22.9


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 28 of 49

picture


56.4 29.4 14.2



59.6 28.5 11.9

Acoustics 36.5 48.5 15




During the first part they felt positive about the size of the screen (73.6%), the fidelity of the

picture (65.8%) and the fidelity of the sound (62.9). They also experienced positively the

movement of the tutor (63.6%) and lip synchronisation (59.4%). Picture (52.7%) and sound

(46.9%) interruptions have been the most annoying aspects of this session.

During the second part of the trial (MPEG-1) the participants felt positive about the size of the

screen (64.7%), the fidelity of the picture (64.7%) and the sound (56.3%). They also expressed

positive feelings about the movement of the tutor (50%) and lip synchronisation (51.6%).

Picture and sound interruptions were not so much of a problem as they were during the 1st part

(22.6% and 32.3% negative views respectively). During the third part of the trial (H.323) again

the size of the screen and the fidelity of the sound and picture was reported positive (58.2%,

49.8% and 56.4% respectively). As in the previous parts the transmission of the movements of

the tutor and the lip synchronisation were positive (less lip synchronisation this time 47.6%). In

this part there have also been picture interruptions (24.1%) but the sound was reported as the

major problem (31.2%).

Diagrams No. 1 and No.2 present the overall opinion of the participants as far as the technical

part is concerned based on the average of their answers to the technical features questions. The

results are contradictory to the straightforward answer of the participants to the open-ended

question calling for their favourite stream.

In the open-ended questions the participants expressed their preference for the second part of

the trial (MPEG-1 – 65%) over the third (H323 – 27.5%) and the first part (MPEG-2 – 7.5%).

We have tried to explain this contradiction using other statistical techniques, like chi-square.

The results of the further analysis are presented in paragraph C4.

13,6

35,4

26,3

20,0

4,5

0,0

10,0

20,0

30,0

40,0

V.P. P. A. N. V.N.

Diagram 1 - Scenario Ia, MPEG-2

9,6

35,939,0

11,6

3,9

0,0

10,0

20,0

30,0

40,0

V.P. P. A. N. V.N.

Diagram 2 - Scenario Ia, MPEG-1


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 29 of 49

2nd Trial – Scenario Ib

1st Part – MPEG-4 (1000 Kbps)



The size of the screen 32.5 15 52.5


22 31.7 46.3


61 19.5 19.5



67.5 22.5 10

Acoustics 55 42.5 20

Sound Interruptions 32.5 42.5 25



2nd Part – MPEG-1 (1000 Kbps)





42.8 16.7 30.6


38.9 16.7 44.4



77.8 13.9 8.3

Acoustics 44.4 33.3 22.2




The quality of the sound (61%), the transmission of the movement of the tutor (67.5%) and lip

synchronisation (68.1%) were the most positive aspects of the first part of the trial according to

the participants' views. The quality of the picture and the size of the screen were considered as

the most negative aspects of this part (46.3% and 52.5% respectively).

During the second part we encountered positive views from the participants concerning most

features (size of the screen 55.5%, transmission of the movements of the tutor 77.8%, lip

synchronisation 48.2% were the most positive. The fidelity of the sound has been reported poor

in this part (negative 44.4%)

Diagrams No. 3 and No 4 refer to the average of the opinions of the participants concerning the

technical features of the second trial.

The open-ended questions give us a clearer picture about the preference of the participants. In

this trial the second trial the participants clearly showed their preference for the 2nd stream

14,7

35,1

24,721,1

4,4

0,0

10,0

20,0

30,0

40,0

V.P. P. A. N. V.N.

Diagram 3 - Scenario Ib, MPEG-4

8,3

36,6

25,2 24,9

4,9

0,0

10,0

20,0

30,0

40,0

V.P. P. A. N. V.N.

Diagram 4 - Scenario Ib, MPEG-1


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 30 of 49

(MPEG-1 – 81%) over the first (MPEG-4 – 19%), which offered them more positive

experiences as far as the quality of the picture is concerned. More than 60% of those who

preferred the first stream, justified their view referring to the better quality of the sound during

the 1st session.

Once again the transmission of the movements of the tutor and the lip synchronisation was very

positive in both parts of the trial, a fact that has certainly contributed to the instructional

success of the trial.

3rd Trial – Scenario Ic – Desktop Users

1st Part - MPEG 4 (500 Kbps)





52.4 14.3 33.3


58.2 19 23.8

Picture Interruptions: 15 75 10


71.4 28.6 0

Echo 16.7 55.6 27.8

Sound Interruptions 35 25 40


Lip synchronisation 19 42.9 38.1

2nd Part - MPEG 1 (500 Kbps)





57.1 19 23.8


76.2 19 4.8



71.4 23.8 4.8

Echo 38.9 44.4 16.7

Sound Interruptions 55 35 10




VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 31 of 49

During the first part of the trial (MPEG-4) the participants expressed positive feelings about the

size of the screen and the quality of the picture (52.4%), the transmission of movement (71.4%)

and the fidelity of the sound (58.2%). On the other hand they criticised the sound interruptions

(40%) and the lack of lip synchronisation (38.1%).

During the second part they were positive about the quality of the picture (57.1%), the quality

of the sound (76.2%), the transmission of movement (71.4%). Picture and sound interruptions

were not a problem in this part (9.6% and 10% negative aspects respectively).

Diagrams No 5 and No 6, show the average of the views of the participants concerning the

technical features.

This time the results are compatible to the opinions expressed at the open-ended questions,

where we can see that the 2nd session (MPEG-1) satisfied about 85% of the participants over the

first session (15%). Many participants from this group justified their preference in technical

terms, referring to the less demanding computer power of the second stream.

C.3 – Scenic Direction

For the first two trials, the camera of the UoA teleteaching room was used to capture the

movement of the tutor. In the 3rd trial a camcorder was used to pre-record the lecture. This

difference clearly affected the views of the participants as to the scenic aspect of the trials. The

respondents clearly favoured the use of the classroom camera (Ia: 65.7%, Ib: 48.8%, while Ic:

36.4% were positive about the position and the angles of the used cameras), which was able to

follow the movement of the tutor more accurately, without losing its focus in most cases.

The participants have identified several features that must be taken under consideration. As

most significant features / aspects of the scenic setting and direction have been mentioned:

The use of movement during the lecture as opposed to steady shots (overall about

54%). At the same time they prefer smooth movement, which does not take the

camera out of focus (34%).

The interchange of views between the tutor and the students or between the tutor and

the instructional materials (41%). Parallel projection of the tutor and the instructional

material was also welcomed (29%).

5,3

32,836,3

21,2

4,4

0,0

10,0

20,0

30,0

40,0

V.P. P. A. N. V.N.

Diagram 5 - Scenario Ic, MPEG-4

15,7

38,1

32,1

9,7

4,4

0,0

10,0

20,0

30,0

40,0

V.P. P. A. N. V.N.

Diagram 6 - Scenario Ic, MPEG-1


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 32 of 49

Avoidance of a long close-up of the tutor (57%). The respondents opted for more

open plans, which carry the feeling of the classroom spatial arrangements (43%).

The tutor is asked to look at the camera (29%). This is reported to help the

establishment of rapport between the host and the remote classrooms.

The lights of the host classroom must be carefully prepared, so that the tutor is shown

without shadows and his / her expressions can be seen when the camera focuses on

him / her (36%).

The instructional material must be designed in a way that facilitates its transmission

(e.g. well-designed slide shows) (37%).

Bad choices concerning these features can affect negatively the attention of the individuals and

trigger repulsive reactions. In the open-ended questions many respondents justify their negative

feelings against streaming video instruction in terms of "false" (according to their perception)

scenic direction choices (28%).

The participants have identified several features concerning the site where they have been, the

remote classroom. The note that everyone must be able to have a clear view of the used screens,

and the lights must not disturb their attendance (26%).

C.4 - A statistical analysis of the factors affecting the satisfaction of the participants

We have analysed the data of all three trials using chi-square test, to reveal the factors affecting

the satisfaction the participants gained from their experience with streaming video.

We have found that the degree of satisfaction was independent from their personal

characteristics, such as their age (p=0.714), their sex (p=0.072) or their study level (p=0.428). It

was also independent from their degree of familiarity with computers (p=0.667). However we

have found that the degree of satisfaction to be significant for those students who claimed to be

very familiar with the Internet (p=0.002), and those who used it regularly to support their

scientific work (p=0.005) or to communicate via e-mail (p=0.036).

Carrying on the analysis on the mutual influence between the pedagogical and the technical

features of the trials, we have found that there is statistically significant relation between the

degree in which the content of the instruction corresponded to the expectations and the interests

of the participants, on the one hand and several organisational - instructional aspects (p<0.005)

and several technological features (p<0.005) on the other, have affected the degree of

satisfaction of the participants.

As far as the instructional factors are concerned, the statistical analysis showed that the

participants gained more satisfaction when (a) the instructional goals were clearly defined

(p=0.000), (b) the information was presented in a structured manner (p=0.000), and (c) when

the tutors took under consideration the prior knowledge of the participants for the subject

(p=0.000).

When the participants had sat in lectures of the tutor before the streaming video trials this also


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 33 of 49

affected positively their degree of satisfaction from the trials (p=0.037). The gestures and the

facial expressions of the tutor (0.003) had also a positive effect on the degree of satisfaction of

the participants as is the case with their movement during the lecture (p=0.001).

Concerning the technical features, we have found that the influence of the quality of the

streaming video picture and sound was a statistically significant factor of the satisfaction of the

participants. During the first trial the satisfaction of students was negatively affected by sound

problems, such as echo (p=0.039) and noise due to the network (p=0.039), while during the

second trial their satisfaction was negatively affected by interruptions in the picture stream (p=

0.011).

GENERAL SUGGESTIONS – CONCLUDING REMARKS

The distance learning trials have shown that streaming video can be an effective educational

medium as far as the opinion of the involved groups is concerned. Instruction through

streaming video is reported to be a source of positive experience to the learners, although

markedly different than conventional instruction.

On the other hand, the participants have strongly stated their views of what they would expect

in order to improve the educational and communicational effectiveness of the medium:

The technical part is the primary source of negative experience for the learners. They

expect the stream to be consistent, without delays or interruptions, obviously

influenced by the features of television. On the other hand, the participants are ready

to compromise to the quality of the streaming video, as long as it does not disturb the

instructional process. The bottom line from their remarks is that they would like the

technology to be a facilitator rather than a potential source of trouble. Consistent

picture, consistent sound and lip synchronisation play a leading part for the learners to

gain a positive experience.

The scenic direction of the tutorial is also important. The learners expect that the view

they get from the host classroom reveals clearly “where the tutor points”, or “what is

written on the whiteboard”, or “who is talking” or “what is the expression on the

tutor’s face”. They call for interchange of views, avoidance of monotonous closed

plans, movement of the tutor. On the other hand they prefer smooth movements that

do not disturb the focus of the cameras and so make it difficult for them to catch up

with what the tutor is doing. If the scenario of the instruction provides for questions

coming from the attendants at the host classroom, they note that all students must

have access to microphones so they can be heard. Bottom line they opt for a carefully

planned scenic appearance of the instructional process, close to what somebody

would expect from a television show.

The instructional aids should be carefully designed. The slides should be clearly seen

and have live colours. They must carry small and consistent pieces of information.

The learners prefer the slides to be always present on their screens. So they opt for the

parallel transmission of the classroom view and the instructional material.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 34 of 49

The tutors should be adequately prepared for the lecture. They must have their course

carefully structured, preferably in a modular way, which allows them to convey

controlled pieces of information. They must also plan their time very carefully, so that

the lecture fits the time frame of the streaming video (approximately 30 to 40

minutes). Longer classes can be proved very tiring for the participants, affecting their

ability to attend effectively. They must also take care of their movements and

expressions so that the remote viewers can understand and follow them.


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 35 of 49

APPENDIX QUESTIONNAIRE

END USER EVALUATION PLAN

QUESTIONNAIRE / INTERVIEW PROTOCOL

“Dear Colleague,

You have been able to participate in an innovative project, investigating alternative ways of

delivery of lectures / lessons through streaming video, a promising technology which is being

developed for transmitting video and sound on the Internet. The aim of our investigation is to

identify the potential pedagogical advantages and / or the potential pedagogical disadvantages /

weaknesses of the use of streaming video in delivering courses / lectures on the Internet. It will

help the research team to improve the technical, instructional, logistical and pedagogical

aspects of the enterprise and to avoid potentially disturbing features of the attempted innovation

in the future.

We would be grateful if you could participate in this semi-structured interview, according to the

protocol issued by members of our research team. We wish to reassure you that anonymity is

guaranteed and that the research team will inform you on the scientific conclusions of the

ongoing project and the outcomes of this survey.”

Section 1 (Filled - out by the research team)

Scenario:

No

Type of Stream / Session

MPEG-2 Mcast-NTUA

MPEG-1 Desktop WMT-medium Desktop


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 36 of 49

MPEG-4 Desktop MPEG-4 Desktop

MPEG-4 Dial-up WMT-low Dial-up

Thematic of the Course:…………………………………………………………..

………………………………………………………………………………………

Tutor:……………………………………………………………………………….

Date: / /

Section 2 - Identification of the type of the End User and

Personal Data

Sex:

Male Female

Age:

19-22 23-26

27-30 31-33

34-37 38-41

Status of the End User:


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 37 of 49

Status Department University

Undergraduate Student _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Postgraduate Student _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Faculty _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Technical Staff _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Administrative Staff _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Other_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Students:

Year / Semester of study _ __ _ _ _ _ _ __ _ _ _ _ _ __ _ _ _ _ _ __ _ _ _ _ _ _

Staff - Faculty - Others:

Type of work _ _ _ _ _ _ __ _ _ _ _ _ __ _ _ _ _ _ __ _ _ _ _ _ __ _ _ _ _ _ _ _

Years of experience_ _ _ _ _ _ __ _ _ _ _ _ __ _ _ _ _ _ __ _ _ _ _ _ __ _ _ _ _

Site of Attendance:

Remote Classroom

Laboratory Which one? _ _ _ _ _ _ _ _ _ _ _ _

Office

Household Connection Speed? _ _ _ _ _ _ _ _ _ _ _ _


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 38 of 49

Other_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Familiarity with PCs

Mention the 3 software programs you use the most:

1. _ _ _ _ _ _ _ _ _ _ _ _ _ 2. _ _ _ _ _ _ _ _ _ _ _ _ _ _ 3. _ _ _ _ _ _ _ _ _ _ _ _ _

Do you posses any certification in a computer related field?

Yes (Which?) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

No

Familiarity with the Internet

I use of the web:

Daily

Weekly (Times?) _ _ _ _

Monthly (Times?) _ _ _ _

Type of use:

General information gathering


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 39 of 49

Scientific work

Communication via e-mail

Surfing for fun / hobbies

Other _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Familiarity with streaming video

None Little Some Extensive

Section 3 - Pedagogical / Instructional Evaluation

Familiarity with the thematic / content of the lecture / lesson:

None Little Some Extensive

Familiarity with the tutor:

This is his / her first lecture I attend


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 40 of 49

I have attended up to 5 lecture with the tutor

I have attended more than 5 lectures with the tutor

Please tick the box which expresses your view on the following:

Components Affecting Instruction Strongly

Disagree

Disagree Neither

Disagree

Nor

Agree

Agree Strongly

Agree

1. The instructional aims have been

clearly defined

2. The information is presented in an

organized / structured manner

3. The speaker used a variety of

instructional methods and techniques

4. The speaker took into account the

degree of familiarity with streaming

video

5. The speaker took into account the

background knowledge of the audience

6. The speaker used feedback questions

during the lecture / lesson

7. I could interrupt the tutor during the


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 41 of 49

lecture, with the use of available

technology

8. The speaker took into account and took

advantage of the students' questions

9. The time devoted to Q+A and

discussion was adequate

10. The students develop collaboration

during the lesson

11. The speaker encourages the audience

to participate actively

12. Familiarity between the tutor and the

students was accomplished

13. The expressiveness of the tutor

(gestures, facial expressions, etc) has

helped me during class

14. The movement of the tutor (walking,

body movements) facilitated the

attendance of the lesson

15. The instructional material that was

used facilitated the attendance of the

lesson

16. The bulk of information presented was


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 42 of 49

excessive

17. The duration of the lesson was

satisfactory

Tele-teaching through streaming video was up to my expectancies and satisfied my

interests

Not at All A Little A Lot Very Much

Lacking the natural presence of the tutor has affected my attendance of the lesson


The use of technology has overthrown the climate of personal contact which is

usually present in conventional tutorials



VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 43 of 49

During the lecture I had the feeling that the tutor:

Lectured only for those students present in the host classroom

Lectured only for those students who attended from remote classrooms

Lectured for everyone attending independent from his / her location

Section 3 - Pedagogical / Perceptual Evaluation of Technical

Features

A. Video and Sound

How would you describe the contribution of the following technical features of picture

and sound of the streaming video you attended to the development of your interest for the

lecture/lesson?

Size of the streaming video window / projection screen:

Very negative Negative Neutral Satisfactory Positive

Fidelity of the streaming video picture (how close to reality was the streaming

video?)



VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 44 of 49

Sound fidelity (clearness, volume, etc)


Video Stream (were there any interruptions / delays?)


Movement of the streaming video picture of the tutor (from awkward movement to

near natural movement)


Echo



VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 45 of 49

Sound Stream (were there any interruptions / delays?)


Noise (from the network)


Lip Synchronisation

(Is there lip synchronisation between what the speaker says and what is seen on the

video?)


B. Production / Direction / Logistics

How would you describe the contribution of the following features of the production

setting of the streaming video you attended to the development of your interest for the

lecture / lesson?


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 46 of 49

Position of camera / cameras


Use of different angles of view


Zoom / focus / view change


Range used in the lecture shooting (range of the speaker’s movement, background

pictures, object setting)



VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 47 of 49

Section 5 - Open questions calling for the opinion of the End User

Are you satisfied from your attendance of the lecture / lesson through streaming video?

(Please explain)

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………

Please specify your views referring to the technical part / to the instructional part / to the

logistical part?

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 48 of 49

Would you like to attend more lectures / lessons through streaming video? (Please specify

the advantages and / or disadvantages).

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………

What are yours suggestions to the research team in order to improve the quality / the

effectiveness of the medium?

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………………………………………………………………

…………………………………………

In case of use of different streams: Which part of the tutorial was (technologically) more

satisfactory (A, B or C)? Please explain.

…………………………………………………………………………………………

....……………………………………………………………………………………….


VERSION 2

DELIVERABLE # 15.4


COMPANY UoA

DATE 30/01/200

page 49 of 49

………………………………………………………………………………………….

………………………………………………………………………………………….

………………………………………………………………………………………….

………………………………………………………………………………………….

—o0o—