The Transformation of QoS/QoETesting and Benchmarking
Presenter: Eng. Mohamed Hedi Jlassi
Prepared by: Dr. Irina Cotanis
Agenda
2
The transformation……
Network testing evolution with InfovistaTEMS
A view on benchmarking changes
Take away: facts and conclusions;
Agenda
3
The transformation……
Network Testing evolution with InfovistaTEMS
A view on benchmarking changes
Take away
Transformation emerging from the technology shift
Processes Automation & RCA reporting
(Reduce engineers, back office decision making)
Automatic Predictive Analysis & Insightful Recommendations
Decisions and recommendations made based on Multi-Sourced Big Data Analytics and AI, alarms when things go wrong
Support for SON and MDT functionalities (Rel. 16+)
Slice and user centric
AI/ML functionality and evaluation techniques
(AI augmented tools to detect and evaluate network AI actions)
Legacy networks
5G integration
5G slicing / new
verticals support
5G QoS/QoEContext aware
QoE
(AI/ML based per
slice, and per user
type/profile
evaluation and
optimization)
5G planning
5G deployment
New freq and antenna patterns
mmW, mMIMO/3D beamfoming
modeling
New freq, new air interface,
multi-band/mode devices
Support for technology
disruptions (e.g. device and
beam centric network)
Rethink test probes
Rethink test/evaluation procedures
Evolve meaning of
QoS/QoE
Enhance test/evaluation: automate
Big Data Analytics
enabled by data driven networks
5
Transformation emerging from AI/ML
AI Analytics for 5G+
Descript
Diagn
Predict
Prescript
Network&Device
Distributed ML/AI( domain/slice//link optimization)
Centralized ML based
SON
Intelligent Wireless
Communica5tions
AI-enabled
closed loop
network
optimization
5G
5G+
Legacy
(3G/4G/LTE)
Transformation in standards:QoS/QoE measurements, test scenarios, tools
6
Transformation in standards: ITU-T/ETSI trends on AI/ML.
Study Group 12: Performance, QoS, QoEMachine learning based QoE(e.g. P.565)InfoVista VoLTE/Vo5G voice testing sQLEAR
Study Group 12: Performance, QoS, QoEIntelligent network diagnosis(e.g. E.845 )InfoVista Analytics
Study Group 13: Future Networks – IMT 2020Focus Group ML for Future Networks including 5G(ML5G)InfovIstaAugmented Measurements
InfoVista Analytics
Agenda
8
The transformation roots
Network testing evolution with InfovistaTEMS
A view on benchmarking changes
Take away
9
Untangling the complex relationship between jitter, packet
loss and MOS
Jitter (ms) (z axis) during one 5.5s long RTP sequence (x axis)
during a call duration (y-axis)
TEMS network testing solutionsRethinking QoE modeling: machine learning required due to increased complexity of the interdependencies
Use case: sQLEAR Speech Quality machine LEARningConcept, field set-up and performance
10
Reference voice sample
QoE MOS predictor
Network centric, free of device’ impact enabling
cost efficient optimization towards network
issues rather than device’s
Network centric prediction
EVS codec / client info
Temporal distribution / DTX
Jitter, packet loss
RTP/IP packet stream
Bit rate, bandwidth, client behavior / error concealment
POLQA
LEARNING
Reference voice
Sync: Temporal distribution / DTX
P.565 based
Operator validation results
11
Real-time analytics and orchestration
TEMS Investigation TEMS Paragon TEMS Sense
Remotely scripted and controlled data collection
TEMS Pocket
Post-processing and analysis
TEMS network testing solution Automation/real time data feeding
TEMS Director TEMS Discovery
Rules based Root
Cause Analysis
Machine learning based
Predict
Real-time Test
Orchestration
TEMS Director
Edge computing
Real time data streaming
TEMS network testing solutionsRethinking testing: ML based probes and automated RCA towards prediction
1
2
3
4
RF Logs
GPS
External data
sources
(e.g. site info)
TEMS 5G Engineering Field Probes
Unattended probes
Play back
Analytics & RCA
TEMS Discovery
IV-TEMS Edge computing
(real time detection/diagnosis at the UE)
Use case: predictive probes for a Connected Car scenario
Realtime analysis
Classify in which
degree certain
limitations are
affecting the
offered service Machine learning algorithms
Classify/categorize
throughput/latency that adapt to QoE
requirements
Predictive analytics of QoE/QoS performed in the network
Edge, TEMS Measurement Agent inside car module enables car
AI
Edge computing
- Call will most likely be dropped shortly……..
- Likely driving through an area of low radio quality, download of
navigation information for that area recommended…
- Within 3 mile there is a reduced overall QoE that will affect the
autonomous system of the vehicle, video streaming will be stopped
and please pay attention to traffic.
AI
NT field
probes
Analytics
TEMS QoS / QoE testing
and prediction
Bad coverage in 20sBad QoS/QoE
Poor coverage in 30sQoS/QoE impact expected
Coverage Limited QoS/QoE in 10s
Coverage in 30sPoor QoS/QoE
No service in 10s
Found legacy network
Use cases RCA – automated VoLTE root cause determination
14
• VoLTE calls shown on a map
• Without Root Causes there is extensive
manual effort
• Root Causes automatically added for
failed calls
• Root Causes are focussed answers that
help drive recovery actions
• No expert resources needed
Use case RCA – real time events
15
• Data is streamed in real time
• Shown here are service events
• System heartbeat events use exactly the same method
• These can be used for real time
determination of measurement device
location and status
TEMS network testing solutions: Readiness to going challenges
16
New spectrum
(F1/F2 spectrum range)
New technologies
(mMIMO/3D beamforming, mmW, )Co-existence with legacy networks (“hybrid” LTE-NR)
New architecture concepts
(NFV/SDN, slicing)
New design concept
(Beam centric / Human-machine centric)
Technology adaptations
(From TCP to QUIC/UDP for the delivery of new services e.g. VR/AR, Video360)
“Traditional” testing within a transformed network
Use case: LTE-NR co-existence (EN-DC) - NSA
17
LTE + NR
Combined view in TEMS Investigation
Use case: 3D scenario for TRP beam characterization and coverage evaluation
18
UE based coverage parameters (RSRP, RSRQ,
CINR) for reference signals and performance
(throughput) in azimuth and elevation
SSS RP, SSS RQ, CI and Beam Index for strongest scanned CI
Verify overall and cell coverage as well as coverage gaps
Identify strongest beams; beam failures
Agenda
19
The transformation…….
Network testing evolution with InfovistaTEMS
A view on benchmarking changes
Take away
ITU-T/ETSI benchmarking transformation
20
Benchmark ETSI scoring transformation
21
Overall Global Network Performance
Score*
(country level)
Cities
GNPS=
=Sum(Aw*ServicesScore)Roads
Composite (hot spots,
trains)
ServicesScore=
= Sum(KPIScore)
Voice/SMS KPIs
Video streaming KPIs
(YT)
KPI Score =value−Bad limit
Good limit−Bad limit×𝒘𝒆𝒊𝒈𝒉𝒕
Area
Weightings(Aw)
Statistical inference at
country level (bootstrapping technique)
Per
area/topology aggregation
Data KPIs
(web browsing, http, SM)
KPI 2 Score Map
Dynamic adaptive
thresholds per applic/slice
Flexible/scalable
weightings
Transformable QoE
Interactivity QoE
- Real time/latency - Continuity (jitter)- Reliability (packet loss)
Benchmarking QoE/QoS transformation (ETSI)
Voic
e • SetupSuccessRatio
• CallDropRatio
• MOS
• MOS<1.6MOS
• 90th percentile of MOS
• SetUpTime
• SetUpTime>1.5s
• 10th percentile of SetUpTime
Vid
eo s
tream
ing • SuccessRatio
• MOS
• 10th percentile MOS
• Access time
• Access time>10s
Data
• Data
• TransferSuccessRatioDL
• Avg.ThroughputDL
• 10th /90th
percentileThroughputDL
• TransferSucessRatioUL
• Avg.ThroughputUL
• 10th /90th
throughputUL
• Browsing
• SuccessRatio
• Avg.Duration
• ActivityDuration>6s
• SM
• SucessRatio
• Avg.Durtaion
• ActivityDuration>15s
22
AR/Remote surgery
AR/VR/360Video
AR/GamingAR/Automotive
ITU Benchmarking in quest for CrowdSource
• Managed & limited controlled user/device
information (connected mode) geolocated,
passive/active(!)
• Network (RAN, IP) information (device dependent)
• Application information (technology agonistic)
• QoE surveys
Reliability
Score
• User centric traffic/resources for
• Trend detection (against thresholds and/or
historic behavior
• Optimization
• BMing: Geographical/demographic/operators
comparisons
Use Cases/
Applications
Application
layerRAN
layers
User/Device
CS
CS
CS
CS
IP layer
Decoding
IE/KPI/API
• Measure of spatial consistency
• Measure of temporal consistency
• Measure of fluctuations and variability
• Absolute number of measuring values
Agenda
24
The transformation….
Network testing evolution with infovistaTEMS
A view on benchmarking changes
Take away
Take away
• Technology (mmW, beam centric, user centric)
• AI/ML embedded, operational and management
The transformation
Infovista
TEMS network testing evolution
• New services/slices require new QoS/QoE dependencies
• New expected performance require dynamic adaptive quality thresholds and weightings
• New data source such as crowd source require reliability testing
A view on benchmarking
changes
25
ML based QoE / sQLEAR
Automation - RCA VoLTE use case
Predictive probes - connective car scenario
3D scenario
Thank you!www.infovista.com
26