OBJECTIVE VIDEO QUALITY ASSESSMENT METHODS.

DESCRIPTION OF A PARAMETRIC METHOD FOR MEASURING VIDEO QUALITY ON DIGITAL TELEVISION

SIGNALS CODED ON H.264Rafael Sotelo

Facultad de Ingeniería, Universidad de Montevideorsotelo@um.edu.uy

INTRODUCTION

Quality of Experience (QoE) is extremely important in communications services, especially in TVD.

TV operators always aim to deliver the best possiblequality to their viewers.

Improving TV Quality has been one of the inductors for DTV.

However, some processes involved in coding the video signal as well as in the transmission, introduce degradations that can lead to low perceived quality

Our site: http://ingenieria.um.edu.uy/vqi 2

¿HOW TO MEASURE VIDEO QUALITY?

Subjective Methods

– The most accurate way. They involve a number of subjects which documents theirperceptions. The mean opinion is obtained directly through the “MOS” (Mean Opinion Score).

– Disadvantages: expensive, hard to make and impractical for real time applications.

Objective Methods

– Automatic methods that accurately predict perceived video quality (i.e., the MOS), based on objective measurements taken in some part of the system.

3

OBJETIVE METHODS

4

BASIC METRICS

Pixel to pixel differences between original images(previous to compression and transmission) and degraded images (after reception and reconstruction)

[ ]∑∑∑= = =

−=N

n

M

m

T

ttnmytnmx

TMNMSE

1

2

1 1),,(),,(1

MSERMSE=

=

MSELPSNR

2

10log10

5

THEY ARE OFTEN CRITIZISED FOR NOT HAVING DIRECT CORRELATION TO PERCEIVED QUALITY PERCEIVED BY SUBJECTS.

Image evaluation. Left: original images. Centre and Right: degraded images.6

Left: original image. Right: degraded image.Up: “Tiffany”, MSE=165; Centre: “Lago”, MSE=167; Above: “Mandril”, MSE=163

7

PERCEPTUAL METHODSRecently much effort has been devoted to develop new methods that can emulate human visual system, which are able to predict human perception using new objective metrics.

Video Quality Expert Group (VQEG).

Content based methods:

– Full Reference

– Reduced Reference

– No Reference

Parametric methods

– Bit rate

– Frame rate

– % packet loss… 8

PERCEPTUAL METHODS

Encoder Transmission(Main

Channel)

Decoder

NR Model

RR Model

FR Model

Auxiliar Channel

Extraction of Key

Charactistics

Transmissiondegradations

Encodingdegradations

DegradedVideo Output

Original Video Input

Qualityestimation

Qualityestimation

Qualityestimation

ITU has standardized some FR and RR models. However, NR models haven’t been standardized yet for TV.

9

PARAMETRIC METHODS

Joskowicz, J; Sotelo, R; López Ardao, J. C; Towards a General ParametricModel for Perceptual Video Quality Estimation, IEEE Transactions onBroadcasting, December 2013, Volume 59, Issue 4, pp. 569- 579

Chen, Y. ; Wu, K. ; Zhang, Q.; From QoS to QoE: A Survey and Tutorial on State of Art, Evolution and Future Directions of Video Quality Analysis, IEEE Communications Surveys & Tutorials, October 2014

10

A PARAMETRIC MODEL FORDIGITAL TERRESTRIALTELEVISION CODED WITH H.264

11

MPEG-2 TRANSPORT STREAM PACKET

4 Bytes Header 184 Bytes Payload

12

MPEG-2 TRANSPORT STREAM PACKETSIN IP PACKETS

184 Bytes Paylaod

40 bytesIP Header

4 bytes (MTS Header)

IP (UDP + RTP) Header MTS Payload

Up to seven MPEG-2 Transport Stream packetsin an IP packet

13

MPEG-2 TRANSPORT STREAM PACKETS IN DTV TRANSMISSION

There are many stages for error correction. Transmission is acheived based on TS packets.

Image from ABNT NBR15601

14

PREVIOUS WORK CONCERNING PACKET LOSSINCIDENCE ON PERCEIVED VIDEO QUALITY

Most of the literature consider packet losses in IP scenarios (streaming orIPTV)

There are differences with DTV:

– When an IP packet is lost, seven TS packets are lost

– Delay

– Jitter

– …

In DTV only individual packets are lost.15

COMMON APPROACHES TO CONSIDER THEINCICENCE OF NOISE ON DTV CHANNEL

In DTV the Bit Error Rate (BER) is often used as a parameter relatedto receiver capacity to reconstruct the transmitted signal.

– For example, DVB-T defines Quasi Error Free Reception (QEF) as less thanone uncorrected error per hour, corresponding to a BER = 10-11 at theinput of the MPEG-2 demultiplexer

However, the last stage in a DTV receiver before image decoding and visualization is the Reed-Solomon (RS) decoder

In the case of ISDB or DVB the RS decoder can reconstruct a TS packet of 188 bytes if it lost up to 8 bytes, and in ATSC a maximumof 10 bytes

Note: DMBT uses other error correction scheme.16

COMMON APPROACHES TO CONSIDER THEINCIDENCE OF NOISE ON DTV CHANNEL

Although the quality of the transmission link is often characterized by theBER, this approach is not the best to evaluate a DTV user QoE

When studying losses related to noise in the DTV cannel, it is necessary to focus on degradations experienced by individual TS packet losses.

17

18

19

20

21

PARAMETRIC MODEL SCHEME

Based on ITU-T G.1070 and on previous literature

where Vq is the estimated quality (MOSp), Ic is the quality determined bythe encoding process and Ip is the quality determined by the transmittingprocess

),( pcq IIfV =

pcqp IIVMOS +== 1

Ic and Ip depend on many parameters

,...),,,,(,...),,,(

contentcodecframeratebitratepacketlossfIcontentcodecframeratebitratefI

p

c

==

PARAMETRIC MODEL SCHEME

PARAMETRIC MODEL – COMPUTING IC

+

×

− +− 641

31

5867201

114 cscac

csc

bhl

+

=I

h – image heightl – image widea1 – related to the codecb – bitrates – SADc1 to c6 tuning parameters

PROPOSED MODEL FOR EVALUATING PACKET LOSSINCIDENCE

We try to compute Ip as a function of known parameters– Percentage of packet loss

– Number of bursts

– …

Joskowicz, J; Sotelo, R.; A model for video quality assessment consideringpacket loss for broadcast digital television coded in H.264. International Journal of Digital Multimedia Broadcasting (2014)

25

IP VS PERCENTAGE OF TS PACKET LOSS FOR HDRESOLUTION

Video quality cannot be estimated only by thepercentage of packet losses and the number of burst!

0,0

0,2

0,4

0,6

0,8

1,0

0,0% 1,0% 2,0% 3,0% 4,0%

Ip

Percentage of TS packet loss

HD Uniform loss

HD One Burst 0,1%

HD One Burst 10%

HD Two Bursts 0,1%

HD Two Bursts 10%

HD Three Bursts 0,1%

HD Three Bursts 10%

26

GOP – GROUP OF PICTURES

I B B P B P B B I

Losses in I slices affect much more than P or B slicesLosses in P slices affect more than B slices

27

NEW METRIC

We define a new metric that represents the weightedpercentage of slices loss, pw

I is the percentage of I slices affected (the number of I slicesaffected with respect to the total number of slices in the clip)

P is the percentage of P slices affected

B is the percentage of B slices affected

x1, x2 are two coefficients

BPxIxpw ++= 21

28

IP VS PW (WEIGHTED PERCENTAGE OF SLICES LOSS)

Ip has a strong correlation with pw, when adequate values for k, x1, x2 are selected

0,0

0,2

0,4

0,6

0,8

1,0

0,0% 20,0% 40,0% 60,0%

Ip

Percentage of weighted Slices loss

HD Uniform loss

HD One Burst 0,1%

HD One Burst 10%

HD Two Bursts 0,1%

HD Two Bursts 10 %

HD Three Bursts 0,1%

HD Three Bursts 10%

wp kp

I+

=1

1

29

MODEL VALIDATION

1

2

3

4

5

1,00 1,50 2,00 2,50 3,00 3,50 4,00 4,50 5,00

MO

Sp

Subjective MOS

Model validation

Pearson Correlation PC =0,92RMSE =0,46

MOSp vs MOS for the first set of subjective tests

MODEL VALIDATION

100 clips in HD format and 100 clips in SD format, 10 seconds long. Recorded from real DTV transmission from two stations in Montevideo.

The dispersion between subjective ratings and those predicted bythe model where calculated. The same value for the coefficientswere used as in the calibration model.

31

1

2

3

4

5

1 2 3 4 5

MO

S

MOSp

Validación del modelo

MODEL VALIDATION

Pearson Correlation PC =0,81RMSE =0,80

MOSp vs MOS for the second set of subjective tests subjetivas

OBJECTIVE VIDEO QUALITY ASSESSMENT METHODS.

DESCRIPTION OF A PARAMETRIC METHOD FOR MEASURING VIDEO QUALITY ON DIGITAL TELEVISION

SIGNALS CODED ON H.264Rafael Sotelo

Facultad de Ingeniería, Universidad de Montevideorsotelo@um.edu.uy