Embed Size (px)
Citation preview
ITU Workshop on “Performance, QoS and QoE of Emerging Networks and ServicesAthens, Greece, September 2015
QoE Evaluation and Enforcement Framework for Internet Services
Marcus Eckert, Thomas M. KnollResearch Assistant, TU-Chemnitz
Contents
• Motivation
• Architecture Overview
• Architecture Components
• QMON – QoE Monitoring
• QRULE – QoE Policy and Rules
• QEN – QoE Enforcement
• Results
• Summary
Page 2
Motivation
• Experienced quality of Internet services is crucial for customer satisfaction
• QoE monitoring and enforcement is thus required for business success• Aim: application specific differentiated handling of traffic flows for major
Internet services• 3GPP standard based procedures using dedicated bearers are hardly
used today• Default bearer differentiated flow handling is missing
Improved QoE measurement and enforcement framework required
ISAAR Framework (ISAAR = Internet Service quality Assessment and Automatic Reaction)
ISAAR augments existing QoS functions by flow based network centric QoE monitoring and enforcement functions
Page 3
Architecture Overview
• Modular service specific QoE management architecture
• 3 functional components:• QoE Monitoring (QMON) – flow detection and measurement, • QoE Rules (QRULE) – policy rules and permission checking and • QoE Enforcement (QEN) – respective flow manipulation
• Interworking with existing QoS mechanisms• 3GPP PCC• Priority marking (DiffServ, Ethernet prio, MPLS prio)• Proprietary router QoS support (queueing, scheduling, shaping)• SDN based QoS support (e.g. through OpenFlow Action Sets)
Page 4
Architecture Overview
Page 5
Architecture Components – QMON
QMON operation• Flow classification
• With and without DPI• Centralized / distributed• With SDN match and action rules
• Flow capturing• SDN support to tee out flows
• Flow Monitoring• Application specific KPI calculation• Standardization: G.102y
QMON output• Flow Information and QoE estimation
• currently implemented: “Video QoE estimation”
Page 6
Measurement Procedure
Measurement at end device• Most precise• Firmware / device specific• User involvement• Tampering possible
Measurement within operator network• User and device independent• End device (buffer) model required estimation• Reliable results at scale• Challenges: constant changes in video streaming (encoding + media
container formats) MPEG DASH
Page 7
Measurement Procedure
Flow Detection and Classification• Deep Packet Inspection (DPI)• Built-in or external using
3GPP PCC Gx interface
Video Quality Measurement• TCP = reliable transport
no longer fine grained pixel and block structure errors
• Video stall events, duration and inter-stall timing is important
• Buffer fill level estimation asmeasurement result
Page 8
Measurement Procedure
Buffer fill level estimation
(exact method)• Difference between TCP segment
timestamp and playout timeencoded in video data within thesegment
• Each segment is traced and processed• Use TCP ACKs to increase precision
(segment loss, RTT measurement)• Buffer model required for fill level estimation
(initial buffering, re-buffering and play-outthresholds)
• Buffer depletion / stall events, re-buffering times and inter-stall timingas measurement results
Page 9
Measurement Improvements (speed & accuracy)
Buffer fill level estimation
(estimation method)• Speed-up by chunk based throughput like measurement to avoid
decoding every packet (“jump through the stream” method)• Full header decoding needed and limited to suitable formats e.g. MP4
• Variable look-up interval trade off between processing speed-up and accuracy
• Loss of fill level estimation precision especially when high delays or even losses occur due to congestion
Header Chunk 1 Chunk 2 Chunk 3 ... Chunk i Chunk i+1
MP4
Page 10
Measurement Improvements (speed & accuracy)
Buffer fill level estimation
(combined method)• Automatic switching between exact and estimation mode of operation
to gain the speed-up during good times and to keep the precision during bad networking conditions.
• There is hardly any difference between the exact and the combined method result.
• However, the processing load increases (speed-up decreases) for bad case video streaming conditions
Page 11
MOS calculation for Video QoE
• Mean Opinion Score (MOS) derived from P.862 (ITU-T: Perceptual evaluation of speech quality = pesq)
• 0 = worst quality / 4.5 = highest quality• Assumption: initial 4.5 decreased by negative impact (NI) factor
Initial buffering (e.g. 10s) does not raise NI
• Each following stall decreases the MOS; follows an e-function (exponential function)
•
where x = # of stall events
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 150
0.51
1.52
2.53
3.54
4.55
stalling events
MO
S
Page 12
• Example video with 5 stalling events• Resulting in a bad MOS value
0 9 18 27 36 45 54 63 72 81 90 99 108 117 126 135 1440
0.51
1.52
2.53
3.54
4.55
time
MO
S
MOS calculation for Video QoE
Page 13
MOS calculation for Video QoE
0 7 14 21 28 35 42 49 56 63 70 77 84 91 98 1051121191261331401470
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
time
MO
S
• Memory effect also influences the negative impact of each stall (D1) as well and dampens the impact the longer the play-out has run smoothly before
Page 14
Architecture Components – QRULE
QRULE input• Flow information and corresponding QoE
estimation from QMON
QRULE operation• Mapping input flow to service flow classes • Check whether QoE enhancement is allowed by
general operator policy• Determine Per Flow Behaviour (PFB) based on
the Enforcement Database of QEN
QRULE output• PFB specific commands for QEN for 3GPP PCC
triggering and/or marking, shaping, dropping and even (SDN/LSP) path selection
Page 15
Architecture Components – QRULE / PFB Determination
Example: PFB commands for marking rules and SDN flow based path
selection in high contention situations
Page 16
0
10
20
30
40
50
60
70
0 50 100 150 200
buff
ered
vid
eo ti
me
in s
packet time in s
LTE Test 720p
Buffer LevelThreshold 1Threshold 2
normal priority
high priority
low priority
Architecture Components – QRULE / PFB Example
EF or equivalent class marking
CS5 marking
BE/LE marking
Page 17
Architecture Components – QEN
QEN input• Flow information and QRULE action
command set
QEN operation• Register enforcement capabilities• Execute flow manipulation via:
• 3GPP – PCC (PCRF / PCEF)• IETF & IEEE priority marking• Automated router configuration
with vendor specific QoS capabilities and settings
• SDN capabilities for marking and Traffic Engineering (TE)
• Granularity: per-flow or per-class• PFB & class PHB / flow & class TE
Page 18
Architecture Components – QEN
QEN flow manipulation options• 3GPP – PCC (PCRF / PCEF)
• QCI marking and/or dedicated bearer setup• IETF & IEEE priority marking
• IP Diffserv, Ethernet priority, MPLS traffic class prioritywithout the need to change the configuration of network elements
• Synchronized inside/outside GTP tunnel & IPSec tunnel marking• Automated router configuration with vendor specific capabilities and settings
• Cisco / Juniper specific router configuration with flow-specific rules for scheduling, shaping, dropping as well as path (LSP) selection
• SDN capabilities for marking and TE• marking via OpenFlow switch action list configuration• flow-specific traffic engineering (LSP selection or flow-specific forwarding
paths)
Page 19
Results
Demonstrator setup additionally to the field trials
• Field trials using packet traces at SGi interface of an operator• Demonstrator Lab setup using 2 Laptops• Online Buffer fill level estimation and MOS calculation
Page 20
Results
Results for
exact vs.
estimation vs.
combined method
of operation
estimation interval stepping
processing time
# re-buffering events
re-buffering
timeBoth algorithms - good case video
Human - 0 0 sExact 6 s 0 0 s
10 packets 3 s 0 0 s50 packets 3 s 0 0 s
100 packets 3 s 0 0 s150 packets 3 s 0 0 s250 packets 3 s 0 0 s
Estimation algorithm - bad case videoHuman - 10 58 sExact 12 s 10 56,6 s
10 packets 6 s 10 56,0 s50 packets 6 s 10 54,4 s
100 packets 6 s 10 53,7 s150 packets 6 s 9 51,3 s250 packets 5 s 6 49,1 s
Combined algorithm - bad case videoHuman - 10 58 sexact 12 s 10 56,6 s
10 packets 8 s 10 56,6 s50 packets 8 s 10 56,6 s
100 packets 8 s 10 56,6 s150 packets 8 s 10 56,6 s250 packets 7 s 10 56,6 s
Page 21
Results
Results for exact vs. combined method of operation
Good case video Bad case video
Page 22
Results
Results for buffer fill level to
MOS QoE calculation
• Each stall is sharply impacting theexperienced quality (MOS score)
• Re-buffering times gradually impactthe MOS score
• Periods of smooth play-out lead toslight MOS recovery
• Full recovery is possible if only a few stalls occur and a long smooth play-out follows
Page 23
0 9 18 27 36 45 54 63 72 81 90 99 1081171261351440
0.51
1.52
2.53
3.54
4.55
time
MO
S
Summary
• ISAAR addresses QoE management for Internet based services
• 3 components QMON, QRULE, QEN to monitor and manipulate flows
• Location aware service flow observation and steering • Automated network based QoE estimation is feasible and produces
accurate results in terms of MOS score calculations and underlying buffer fill level estimations.
• Aware of 3GPP standardized PCC • Interworking with PCRF/PCEF• 3GPP interfaces are supported (Sd, UD/Sp, Rx, Gx/Gxx)
• ISAAR is also able to work independently of 3GPP QoS functionality
Page 24
