H. W. Gierlich - ITU. W. Gierlich HEAD acoustics GmbH ... 11 participating companies, test lab HEAD acoustics • 4th SQTESQTE June 2006 , ... EC implementation •Double talk

ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006

International Telecommunication Union

ITU-T

Speech Quality Testing for Speech Quality Testing for

VoIPVoIP Terminals and Terminals and

Gateways: Gateways:

Input from ETSI “Input from ETSI “PlugtestPlugtest” ”

Speech Quality Test Events Speech Quality Test Events

H. W. GierlichHEAD acoustics GmbH

ITU-T Workshop on “End-to-End QoE/QoS“Geneva, 14-16 June 2006 2

ITU-T

Overview

o Introduction - 1st, 2nd and 3rd

SQTE

oDiscussion of Results

•Gateways

•IP phones

o Summary & Outlook


ITU-T

Overview


SQTE


•Gateways

•IP phones

o Summary & Outlook


ITU-T

Introduction

The ETSI SSpeech QQuality TTest EEvents

• 1st SQTE October 2000 organized and hosted by the ETSI “Bake-off Service”

6 participating companies

test labs HEAD acoustics, Deutsche Telekom

• 2nd SQTE April 2002organized and hosted by the ETSI “Plugtests Service”

5 participating companies

test labs HEAD acoustics, Deutsche Telekom

• 3rd SQTE June and September 2004, “twin event” EU, US organized by the ETSI “Plugtests Service” and HEAD acoustics, hosted by HEAD acoustics

11 participating companies, test lab 11 participating companies, test lab HEAD acousticsHEAD acoustics

•• 44thth SQTESQTE June 2006June 2006, EU – just ongoing


ITU-T

The Goal

Evaluate all conversational aspects• Listening speech quality (“status quo”)

• Detailed parameter tests

• Echo measurements, EC implementation

• Double talk performance tests

• Quality of background noise transmission

In order to•• Compare different implementationsCompare different implementations

•• Document the state of the arte of technologyDocument the state of the arte of technology

•• Give input to standardizationGive input to standardization

•• Help companies to optimize their implementationsHelp companies to optimize their implementations


ITU-T

Test Setup Gateways

Condition Packet Loss (Equal) Additional Delay1 Delay Variation

1a 0 0 No

2a 1% 0 No

3a 2% 0 No

4a 3% 0 No

5a 5% 0 No

6a 1% 50 ms 20 ms (2)

Gateway

IP

E1

NISTNet

Packet MONITOR

E1

Packet Loss Delay

Output

HEAD acoustics Test System AACCQQUUAA

Input

Gateway

IP IP

ISDN DSS1 ISDN DSS1

PBX

Condition Packet Loss (Equal) Additional Delay1 Delay Variation

1b 0 0 No

2b 5% 0 No

3b 0 50 ms 20 ms (2)

4b 5% 50 ms 20 ms(2)


ITU-T

Test Setup IP-Phones

IP

NISTNet

Packet MONITOR

E1

Packet Loss Delay

Output

HEAD acoustics Test System AACCQQUUAA

Input

Gateway

IP

ISDN DSS1

PBX

IP

IP Terminal

(Phone or PC)

Condition Packet Loss (Equal) Additional Delay1) Delay Variation

1c 0 100 ms No

2c 0 100 ms 20 ms(2)

3c 1% 100 ms No

4c 1% 100 ms 20 ms(2)

5c 3% 100 ms No

Condition Packet Loss (Equal) Additional Delay1) Delay Variation

1d 0 100 ms No

2d 3% 100 ms No

3d 0 100 ms 20 ms(2)

4d 3% 100 ms 20 ms(2)


ITU-T

Overview


SQTE


•Gateways

•IP phones

o Summary & Outlook


ITU-T

Listening Speech Quality PESQ P.862 (MOSPESQ P.862 (MOS--LQO)LQO)

1

2

3

4

5

0%, 0ms,

VAD on

0%, 0ms 1%, 0ms 2%, 0ms 3%, 0ms 5%, 0ms 1%, 20ms,

r=0.9 (fix

delay 50ms)

1%, 20ms,

r=0.5 (fix

delay 50ms)

network conditions

MO

S-L

QO

0

100

200

300

400

500

600

de

lay

/ m

s

PESQ delaymax. MOSmax. MOS--LQOLQO

minmin

averageaverage

max. max. delaydelay

minminaverageaverage

G.711


ITU-T

Listening Speech Quality TOSQA2001 (TMOS)TOSQA2001 (TMOS)

1

2

3

4

5

0%, 0ms, VAD

on

0%, 0ms 1%, 0ms 2%, 0ms 3%, 0ms 5%, 0ms 1%, 20ms,

r=0.9 (fixdelay 50ms)

1%, 20ms,

r=0.5 (fixdelay 50ms)

network conditions

TM

OS

0

100

200

300

400

500

600

de

lay

/ m

s

TOSQA delay

G.711


ITU-T

Listening Speech Quality (MOS(MOS--LQO vs. TMOS)LQO vs. TMOS)

1

2

3

4

5

0%, 0ms,

VAD on

0%, 0ms 1%, 0ms 2%, 0ms 3%, 0ms 5%, 0ms 1%, 20ms,

r=0.9 (fix

delay 50ms)

1%, 20ms,

r=0.5 (fix

delay 50ms)

network conditions

TM

OS

, M

OS

-LQ

O

PESQ TOSQA

For further SQTE results (G.729, G.723) see:For further SQTE results (G.729, G.723) see:

AnonymizedAnonymized Report Report –– “Gateways” “Gateways”

SQTE Results IP Phones: SQTE Results IP Phones:

AnonymizedAnonymized Report Report –– “IP Phones” “IP Phones”


ITU-T

Packet Loss Concealment

Highly influencing MOS score - optimization

criteriaTest Test signalsignal (5s)(5s) •• ITUITU--T T RecommendationRecommendation P.501 P.501

•• periodicalperiodical repetitionrepetition of a of a voicedvoiced soundsound

•• pitchpitch frequencyfrequency approxapprox. 330 Hz . 330 Hz

Relative ApproachRelative Approach •• hearinghearing modelmodel basedbased forwardforward estimationestimation

•• comparisoncomparison to to actualactual signalsignal

•• displaysdisplays ““estimationestimation errorerror“ “

Cross Cross correlationcorrelation

vs. Timevs. Time•• analysisanalysis of of signalsignal phasephase

•• periodicalperiodical patternpattern ((approxapprox. 3 ms). 3 ms)


ITU-T

Packet Loss Concealment

U-V

-0.3

-0.2

0.2

0.3

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

f-

Hz

300

2k

4k

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

cp/cPa0 2.5 5 10 12.5 15

t-s

-3m

3m

5m

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

%-100 -75 -50 -25 25 50 75 100

U-V

-0.5

-0.3

0.3

0.5

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

f-

Hz

300

2k

4k

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

cp/cPa0 2.5 5 10 12.5 15

t-s

-3m

3m

5m

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

%-100 -75 -50 -25 25 50 75 100

U-V

-0.3

-0.2

0.2

0.3

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

f-

Hz

300

2k

4k

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

cp/cPa0 2.5 5 10 12.5 15

t-s

-3m

3m

5m

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

%-100 -75 -50 -25 25 50 75 100

U-V

-0.4

-0.2

0.2

0.4

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

f-

Hz

300

2k

4k

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

cp/cPa0 2.5 5 10 12.5 15

t-s

-3m

3m

5m

t/s0.5 1 1.5 2 3 3.5 4 4.5 5

%-100 -75 -50 -25 25 50 75 100

MOSMOS--LQO LQO

5% packet 5% packet lossloss

3.63.63.73.7

3.13.1 3.33.3


ITU-T

VAD and Comfort Noise

IP

Packet Loss

Delay

GatewayGateway

IP

PBX PBX PBX PBX ISDN

Simulator

ISDN

Simulator

ISDN

Simulator

ISDN

Simulator

NISTNet

Voice Activity Detection

silence suppression

silence packets

comfort noise generation


ITU-T

VAD und Comfort Noise

Level vs. time Manual(5.0 ms) L/dB[V]

-70

-65

-60

-55

-50

-45

-40

-35

-30

t/s4.5 4.6 4.7 4.8 4.9 5.1 5.2 5.3 5.4 5.5

Level vs. t ime Manual(5,0 ms) L/dB[V]

-70

-65

-60

-55

-50

-45

-40

-35

-30

t/s4.5 4.6 4.7 4.8 4.9 5.1 5.2 5.3 5.4 5.5

Level vs. t ime Manual(5,0 ms) L/dB[V]

-70

-65

-60

-55

-50

-45

-40

-35

-30

t/s4.5 4.6 4.7 4.8 4.9 5.1 5.2 5.3 5.4 5.5

Level vs. time Manual(5,0 ms) L/dB[V]

-70

-65

-60

-55

-50

-45

-40

-35

-30

t/s4.5 4.6 4.7 4.8 4.9 5.1 5.2 5.3 5.4 5.5


-70

-65

-60

-55

-50

-45

-40

-35

-30

t/s4.5 4.6 4.7 4.8 4.9 5.1 5.2 5.3 5.4 5.5

voiced sound,

decreasing and increasing

level vs. time

test test signalsignal levellevel

measuredmeasured signalsignal levellevel

activationactivation thresholdsthresholds, “, “hysteresishysteresis““

adaptive adaptive comfortcomfort noisenoise


ITU-T

VAD und Comfort Noise II


-80

-70

-60

-50

-40

-30

-20

-10

0

t/s5 6 7 8 9 10


-80

-70

-60

-50

-40

-30

-20

-10

0

t/s5 6 7 8 9 10

Level vs. time Manual(5.0 ms) L/dB[V]

-80

-70

-60

-50

-40

-30

-20

-10

0

t/s5 6 7 8 9 10


-80

-70

-60

-50

-40

-30

-20

-10

0

t/s5 6 7 8 9 10


-80

-70

-60

-50

-40

-30

-20

-10

0

t/s5 6 7 8 9 10

Realitsic noise scenario

(students cafe)

noisenoise contrastcontrast

quicklyquickly adaptingadapting comfortcomfort noisenoise

comfortcomfort noisenoise levellevel slightlyslightly tootoo high high


ITU-T

Setup for Echo Canceller Testing

IP

Packet Loss

Delay

GatewayGateway

IP


Simulator

ISDN

Simulator

ISDN

Simulator

ISDN

Simulator

InRCV

OutRCV

InSND

OutSND

NISTNet

USB

MFE VI

Receive direction

Send direction

(Echo (Echo pathpath

realizedrealized in in

MFE VI) MFE VI)

EC under test

far endnear end

Test system

ACQUA„„infinite“infinite“

ERL 40 dB and ERL 40 dB and

ERL 6 dB ERL 6 dB

EC off


ITU-T

Double Talk Performance

IP

Packet Loss

Delay

GatewayGateway

IP


Simulator

ISDN

Simulator

ISDN

Simulator

ISDN

Simulator

InRCV

OutRCV

InSND

OutSND

NISTNet

USB

MFE VI

Testsystem

ACQUA„„infinite“infinite“

ERL 40 dB andERL 40 dB and

ERL 6 dBERL 6 dB

EC under test EC off


ITU-T

Double Talk Performance 6 dB ERL

U/V

-0.15

-0.1

-50m

0

50m

0.1

0.15

t/s14 14.5 15 15.5 16.5 17 17.5

U/V

-0.15

-0.1

-50m

0

50m

0.1

0.15

t/s14 14.5 15 15.5 16.5 17 17.5

U/V

-0.15

-0.1

-50m

0

50m

0.1

0.15

t/s14 14.5 15 15.5 16.5 17 17.5

U/V

-0.15

-0.1

-50m

0

50m

0.1

0.15

t/s14 14.5 15 15.5 16.5 17 17.5

DT_VAR_S.DAT p/Pa

-1.25

-1

-0.75

-0.5

-0.25

0

0.25

0.5

0.75

1

1.25

t/s0 5 10 20 25 30 echo echo componentscomponents

clippingclipping


ITU-T

Summary Results: „Gateway Pie“

conversationalconversational aspectsaspects

likelike echo echo behaviourbehaviour, ,

double talk double talk performanceperformance

and and backgroundbackground noisenoise

transmissiontransmission

listening speech quality


ITU-T

Summary Results: „Gateway Pie“ (ITU-T P.505)

conversational aspects

like echo behaviour,

double talk performance

and background noise

transmission

listening speech qualityEachEach ““slideslide“ “ representsrepresents

oneone parameterparameter


ITU-T

Summary Results: „Gateway Pie“(ITU-T P.505)

conversational aspects

like echo behaviour,

double talk performance

and background noise

transmission

listening speech qualityAssumptionAssumption: : IndependencyIndependency ofof

parametersparameters, ,

no no interactioninteraction

aspectsaspects


ITU-T

Summary Results: „Gateway Pie“(ITU-T P.505)


likelike echo echo behaviourbehaviour, ,

double talk double talk performanceperformance

and and backgroundbackground noisenoise

transmissiontransmission

listeninglistening speechspeech qualityqualityMinimum Minimum requirementrequirement

(red (red circlecircle) ) representsrepresents

““averageaverage performanceperformance“ “

of 3of 3rdrd SQTESQTE

……oror

recommendedrecommended

numbersnumbers


ITU-T

Example Results (4 of 10)

““balancedbalanced“ “

implementationimplementation

jitterjitter bufferbuffer

performanceperformance G.711,G.711,

significantsignificant

disturbancesdisturbances

G.711 PLCG.711 PLC

EC EC insertsinserts high BGN high BGN

modulationmodulation (NLP/CN) (NLP/CN)

BGN BGN modulationmodulation

causedcaused byby VAD/CN VAD/CN

violationviolation of of

requirementrequirement onlyonly

causedcaused byby CN CN levellevel(echo (echo attenuationattenuation high high enoughenough))


ITU-T

Overview


SQTE


•Gateways

•IP phones

o Summary & Outlook


ITU-T

Testing IP Phones @ 3rd SQTE

Tests in handset and hands-free mode

For handset

• Frequency response,

…

• TMOS using

TOSQA2001

• Echo measurements

• Double talk

performance

• Quality of

background noise

transmission

2 N

13 N


ITU-T

Testing IP Phones @ 3rd SQTE

Receiving

frequency

responses

and TMOS

L/dB[Pa/V]

-30

-20

-10

0

10

20

30

f/Hz240 300 400 500600 1200 1600 2400 4000

L/dB[Pa/V]

-40

-30

-20

-10

0

10

20

f/Hz240 300 400 600 1200 1600 2400 4000

L/dB[Pa/V]

-30

-20

-10

0

10

20

30

f/Hz240 300 400 600 1200 1600 2400 4000

L/dB[Pa/V]

-30

-20

-10

0

10

20

30

f/Hz240 300 400 500600 1200 1600 2400 4000

3.1 TMOS 2N

3.6 TMOS 8N

3.7 TMOS 13N

3.2 TMOS 2N

3.7 TMOS 8N

3.9 TMOS 13N

2.9 TMOS 2N

3.5 TMOS 8N

3.8 TMOS 13N

3.1 TMOS 2N

3.8 TMOS 8N

4.0 TMOS 13N


ITU-T

Listening Speech Quality TOSQA2001 (TMOS)TOSQA2001 (TMOS)

1

2

3

4

5

sending 2N 8N 13N 0%, 0ms 0%, 20ms,

r=0.9

1%, 0ms 1%, 20ms,

r=0.9

3%, 0ms 1%, 20ms,

r=0.5

network conditions

TM

OS

0

100

200

300

400

500

600

700

800

de

lay

/ m

s

TOSQA delay

G.711, sending & receiving direction


ITU-T

Double Talk Test Hands-Free Terminals

Level vs . t ime Manual(5,0 ms) Ch.1 - Ch.2 L/dB

-60

-50

-40

-30

-20

-10

0

10

t/s0.25 0.5 0.75 1 1.25 1.75 2 2.25 2.5

Level vs. time Manual(5,0 ms) Ch.1 - Ch.2 L/dB

-60

-50

-40

-30

-20

-10

0

10

t/s0.5 0.75 1 1.25 1.75 2 2.25 2.5 2.75


-60

-50

-40

-30

-20

-10

0

10

t/s0.25 0.5 0.75 1 1.25 1.75 2 2.25 2.5


-60

-50

-40

-30

-20

-10

0

10

t/s0.25 0.5 0.75 1 1.25 1.75 2 2.25 2.5

Reference

Double talk test:

Attenuation

range

in sending

direction

(P.340)


ITU-T

Summary Results: “IP-Phone Pies“(ITU-T P.505)


handsethandset

-- echo, double talk, echo, double talk,

BGN BGN transmissiontransmission --


handshands--freefree

-- echo, double talk, echo, double talk,

BGN BGN transmissiontransmission --

handset

listening speech quality


ITU-T

Summary Results: “IP-Phone Pies“

““balancedbalanced“ “

handsethandset

implementationimplementation……

““balancedbalanced“ “ handsethandset

ImplementationImplementation……


ITUITU--T P.340 T P.340

““typetype 3“ 3“ implemimplem..



““typetype 3“ 3“ implemimplem..

G.729 and G.729 and

handshands--freefree

notnot testedtested



““typetype 3“ 3“ implemimplem.,.,

ViolationViolation causedcaused

byby noisenoise levellevel

(echo (echo attenuationattenuation

high high enoughenough))


ITU-T

Some quotes from the 3rd SQTE event:

o Michael Metzger, Executive Director of Marketing, Mindspeed Technologies:• "... bringing together the world leaders in VoIP technology in a co-operative

and neutral environment, this prestigious event continues to improve thequality of VoIP...„

o Peter Faxel, member of the executive board, AVM;• "... The ETSI event has helped us towards achieving our goals, and given us

valuable information about the market trend and technologicaldevelopments."

o Daniel Hartnett, Infineon Technologies:• Plugtests(tm) service of ETSI is an invaluable exercise for a company that

sees voice quality as a key element for the success of our telephoneterminal product families. Infineon will continue to "The VoIP SpeechQuality Test Event (SQTE) organised by the work closely with HEAD acoustics and ETSI to benchmark their products for voice qualityexcellence"

Overall Overall qualityquality of of thethe eventevent: Score 4.6 out of 5: Score 4.6 out of 5

Documents

H. W. Gierlich - ITU. W. Gierlich HEAD acoustics GmbH ... 11 participating companies, test lab HEAD acoustics • 4th SQTESQTE June 2006 , ... EC implementation •Double talk