ETSI STQ, Taiwan Workshop, February 13th, 2006 1 Recent improvements in transmission quality assessment : Background noise transmission Results of STF

ETSI STQ, Taiwan Workshop, February 13th, 2006

1

Recent improvements in transmission quality assessment :

Background noise transmission

Results of STF 273, STF 294

Jean-Yves Monfort, France Telecom

ETSI TC STQ Chairman


2

STFs

STF : Special Task Force Founded by ETSI (STF 273) and e-Europe (STF 294) Several Experts involved in a project new inputs in

standards

STF 273 Speech quality performance in the presence of background noise

A noise data base, freely available

EG 202 396-1 STF 294 Improving the quality of eEurope wideband

speech applications by developing a standardised performance testing and evaluation methodology for background noise transmission

EG 202 396-2


3


Objectives : a close-to-original recording and reproduction system

The background noise recording technique should be– Easy to use – Easy to calibrate – Capable of wideband recording– Available at reasonable costs– Mostly compatible with existing standards and procedures used in telecommunications testing


4


Objectives : a close-to-original recording and reproduction system

The background noise simulation (reproduction) arrangement should be:– easy to set up – usable in typical office type room environments without special acoustical treatment– provide a mostly realistic background noise simulation for all typical background noises occurring in telecommunication applications– easy to calibrate – mostly insensitive to the positioning of (test) objects in the simulated sound field– applicable to all typical terminals used in telecommunication– available at reasonable costs


5

STF 273 : The Recording Setup

Recordings for the setup in office rooms with Artificial heads (ITU-T Rec.

P.58) or Binaural probe microphone

(MIRE technique)

artificial headequalizer right

AH or BMEqualizer right

Artificial Head (AH)

or

Binaural Microphone

worn by a subject (BM)

Upload toData Base

AH or BMEqualizer left


6

STF 273 :The Background Noise Simulation Setup: 4.1 - Technique

Subwoofer

Analyser (1/2-channel)

2m 2m

2m 2m

Playbackfrom

database

Terminal under test


7

STF 273 : Comparison of Terminal Performance: Simulated vs. Real

Background Noise

L/dB[V]

-100

-90

-80

-70

-60

-50

-40

-30

f /Hz100 200 500 1000 2000 5000

MobPh1_BGN SP_original.3rdOctave(4096,50.0%)MobPh1_BGN SP_simulated.3rdOctave(4096,50.0%)

L/dB[V]

-100

-90

-80

-70

-60

-50

-40

-30

f /Hz20 50 100 200 1000 2000 5000 10k 20k

Headset_original.3rdOctave(4096,50.0%)Headset_simulated.3rdOctave(4096,50.0%)

L/dB[V]

-100

-90

-80

-70

-60

-50

-40

-30

f /Hz100 200 500 1000 2000 5000

DesktopHFT_original.3rdOctave(4096,50.0%)DesktopHFT_simulated.3rdOctave(4096,50.0%)

L/dB[V]

-100

-90

-80

-70

-60

-50

-40

-30

f /Hz50 100 200 500 1000 2000 5000 10k

WB_HFT_original.3rdOctave(4096,50.0%)WB_HFT_simulated.3rdOctave(4096,50.0%)

Headset

Mobile phone

Desktop HFTWidebandconference system

Green – originalRed - simulated


8

STF 273 Background noise Recording in cars

Instead of an artificial head, two methods for sound pick-up and equalization :

• the microphone of the terminal under test,• a pair of cardioid microphones

Background noise Reproduction in cars


9

STF 294

Improving the quality of eEurope wideband speech applications by developing a standardised performance testing and evaluation methodology for background noise transmission

Objectives : to develop a quality model for wideband speech for new multimedia services (e-health, e-learning, e-bussiness,…)

Program in two phases Phase 1: Network simulation, Subjective tests (in french

and csez langages) on realistic scenarios with background noise. Finalized in end 2006;

• EG 202 396-2 available

Phase 2 : Development and validation of an objective meyhodology for background noise transmission.

Under Progress August 2006


10

Why Background noise is a problem in mostly all situations and conditions?Both networks and terminals involved

In mobile communication background noise is almost always present; Problem emphasized due to the acoustical construction of

mobile terminals. Situation even worse for mobile hands-free terminals and

mobile car hands-free terminals. In multimedia conference and classroom systems

hands-free type terminal with quite high distance between microphone und speaker.

Computer terminals (e.g. in VoIP scenarios) often used as hands-free terminals Background noise may be produced by the computer/handheld

device itself but may also be picked up from the environment such as an open plan office.

Effects of transmission network impairments in packet based communication networks, Wireless LAN,…


11

Scope of the work STF 294

The aim of the work is to find methods for background noise transmission quality evaluation for real life network scenarios with speech present at the near end and at the far end as well as during periods where no speech is present but only background noise is transmitted.

Steps :• Setup and verification of a transmission network simulation

environment using realistic network scenarios for laboratory use;

• Setup of a database containing the relevant transmission network models and traffic patterns that would be used for subjective and objective evaluation;

• Subjective tests forming the basis for objective evaluation procedures, using the background noise database developed in the currently active ETSI STF 273;

• Evaluation and validation of background noise transmission evaluation procedures.


12

Phase 1 Tasks – References for the development of the new objective Method

Phase 1 (Leader France Telecom) Task 1: Transmission network simulation: Telefonica

A transmission network simulation environment needs to be setup, to provide comparable network characteristics and traffic patterns as found in conditions of the real transmission networks.

Task 2: Database for transmission networks and conditions evaluation/simulation Telefonica

A database containing the typical and realistic transmission network scenarios and traffic patterns needs to be created. There are no risks expected in this task, the procedures are known by the experts in this field.

Task 3: Subjective tests, France Telecom (leader), Mesaqin

Subjective test results providing the speech quality parameters in the background noise situation by the individual parameters are to be conducted. This task must be conducted by experts in the field of subjective testing.


13

STF 294

SFT-Project„BackgroundNoise Simulation“- Background noise simulation

setup- Background noise database

STF 273(existing STF-Project)

Task 1, 2eEurope„NetworkSimulation“

Task 3eEurope„SubjectiveTesting“

Task 4eEurope„ObjectiveMethod“

Task 5eEurope„Validation“

Task 6eEurope„FinalStandardization“

Phase 1 Phase 2

ETSI-guide,background Noise

database

ETSI-guide,database with subj.

results

ETSI-guide,network conditions

database

ETSI-guide,draft objective test

method

ETSI-guide, validated objective test

method

final ETSI-guide(s),databases & evaluation

method


14

Task 1 : Transmission network simulation

Evaluation of simulation methodologies and tools Choice of the best simulation methodologies and tools Setup of simulation environment in the laboratories of the STF

experts Validation of the simulation environment The transmission network simulation tool chosen by the

experts of STF 294 is an IP network simulator (namely : NIST Net). The network simulation is supposed to provide comparable network characteristics and traffic patterns as found in conditions of the real transmission networks (in terms of packet loss, delay and delay variation distributions).


15

Task 2 : Database for transmission networks and conditions evaluation/simulation

Definition of the different network scenarios and conditions to be recorded

Preparation of the simulation recordings

Editing and preparing the data base

Description of the data base (STQ document)


16

Steps in Recording Process

Step1.wav

Speech sample database (4320 samples)

Database of noise type/wideband terminal/network impairment combinations

Speech samples with noise and terminal conditions (720 samples)

Speech samples with background noise conditions (80 samples)

Noise-free speech samples (8 samples)

Step 1 :Clean speech sequences recording

Step 2 :Noisy conditions adding

Step 3 : Wideband terminal simulation

Step 4 : Network simulation

Step2.wav


17

Recorded speech samples

For the recordings performed by ETSI STF 294 the following conditions have been used: 48 kHz (16 bit) sampling rate; wave format; and the active speech level has been equalized to -26 dBoV (see ITU-T

Recommendation P.56 [24], no filtering applied). Also for STF 294, the number of samples to have a

representative database large enough for subjective and objective evaluation has been decided: 4 speakers (2 male, 2 female), 8 sentences each; two languages: Czech and French. The length of recordings varies between 24 and 73 s. They contain

neutral sentences of 2-3s length separated by pauses. Speech activity factor varies between 30 and 60 %. This arrangement allows for easy later editing to any required format.

Simulated terminals : Mobile Handset in the street Hands-Free with PC Hands-Free in car


18

Noisy Conditions / Noisy signal Processing

Noisy Conditions 5 background noise conditions selected:

• cafeteria noise;• office room noise;• road noise;• crossroads; and• car noise (car hands‑free at 130 km/h).

Each combination of speech file plus background noise is recorded at two microphone‑loudspeaker positions:

• typical handset microphone position (with loudness ratings adjusted to 7 dB); and

• hands‑free microphone position (with loudness ratings adjusted to 11 dB).

Noisy signal Processing Three parameters

• With and without VAD (Voice Activity Detection)• 2 different noise reduction filters• 2 different noise reduction levels


19

Task 3 : Subjective tests

Preparation of the test material

Conducting experts tests

Evaluation of the experts test results

Preparing test material for naïve listeners tests

Conducting naïve listeners tests (2 languages : French and Czech)

Evaluation of the results

Statistical analysis the test results


20

Listening testsMethod used : according to ITU-T Recommendation P.835

three questions :- perception and judgment of noise level reduction,- perception and judgment of possible distortion on

speech signal,- overall perception of quality.

All these questions must be answered on a discrete 5 level scale.

France Telecom R&DTest Laboratory

Mesaquin Test Lab


21

Exemples of results (1)

MOS distribution and associated 95% CI per noise condition for Czech and French data

C zech

French

Car

-han

dset

Car

-han

ds-f

ree

Cro

ssro

ad-h

ands

et

Cro

ssro

ad-h

ands

-fre

e

Roa

d-ha

ndse

t

Offi

ce-h

ands

et

Offi

ce-h

ands

-fre

e

Caf

eter

ia-h

ands

et

NO ISE

1,0

1,5

2,0

2,5

3,0

3,5

4,0

4,5

5,0

MO

S


22


MOS distribution and associated 95% CI per network condition for Czech and French data (NETWORK I: delay jitter 0ms. packet loss 0%. NETWORK II: delay jitter 10ms. packet loss 1%. NETWORK III: delay jitter 20ms. packet loss 3%)

C zech

French1 2 3

NETW O R K

1,0

1,5

2,0

2,5

3,0

3,5

4,0

4,5

5,0

MO

S


23


MOS distribution and associated 95% CI per NSA condition

No

NS

A

no v

ad_s

moo

th_9

no v

ad_s

moo

th_1

8

no v

ad_s

harp

_9

no v

ad_s

harp

_18

vad_

smoo

th_9

vad_

smoo

th_1

8

vad_

shar

p_9

vad_

shar

p_18

NSA

1,0

1,5

2,0

2,5

3,0

3,5

4,0

4,5

5,0

MO

S


24

What do we also find in EG 202 396-2

Performance parameter descriptions (IP Transmission) Delay (from codec, packetization, network,…) Jitter Packet loss

Parameter interaction and dependences

Wideband speech codecs Characteristics Descriptions


25

What do we also find in EG 202 396-2

Wideband speech codecs Characteristics

• Speech coding algorithm• DTX (discontinuous transmission)• Packet Loss Concealment (PLC)• …

Description of some wideband codecs• ITU-T G.722• AMR WB• …


26

Next Phase – Development, Validation and sharing of he new objective Method

Task 4: Objective methodAn objective method for the quantification of

background noise transmission performance needs to be developed. New methods shall be developed and/or existing methods have to be adapted to the problem.

Task 5: Validation of the new methodThe validation has to be performed independently from

the development of the background noise evaluation method based on unknown data acquired in task 3.

Task 6: Final standardization of the new method

The final standard(s) are drafted and adapted to the results of the previous tasks. An STQ workshop is planned and realized.

Workshop in Praha, November 13-14, 2006


27

Complementary informations

Visit the STF 273 and STF 294 Web sites

http://portal.etsi.org/STFs/STQ/STF273.asp http://portal.etsi.org/STFs/STQ/STF294.asp

You will find :

ToRs of the STF More informations on the work programme Coordinates of the experts Database, results,…


28

Questions ?

Documents

ETSI STQ, Taiwan Workshop, February 13th, 2006 1 Recent improvements in transmission quality assessment : Background noise transmission Results of STF