Embed Size (px)
Citation preview
1
PERCEVAL - Perceptual and Cognitive Quality Evaluation Techniques for
Audiovisual Systems
Ulrich [email protected]
2
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
3
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
4
Multimedia systems rely on compression and encoding to speed up transmission
As a result, media quality goes down and the audio and video streams can become misaligned
Compression reduces the file size by removing redundant or unessential information
Ulrich Reiter – PERCEVAL
5
Perceptual consequences of distortion:
- Face identification- Scene/object recognition- Speech comprehension- Audiovisual speech
integration- Temporal order of events
- Audiovisual temporal integration?
- Long duration quality perception?
Ulrich Reiter – PERCEVAL
6
PERCEVAL project
Ulrich Reiter – PERCEVAL
7
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
8
Do quality expectations decrease over time and with increased involvement in the content?
How fast can users notice quality changes and at which quality level does this happen?
Is the quality level at which the change is noticed similar to the desired quality level?
Questions to be answered
Ulrich Reiter – PERCEVAL
9
Quality assessment methods for long duration AV content
Ulrich Reiter – PERCEVAL
10
SSCQE(Single Stimulus Continuous Quality Evaluation)
Ulrich Reiter – PERCEVAL
11
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
12
Method description
Quality adjustment
Ulrich Reiter – PERCEVAL
13
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
14
Experimental conditions
Ulrich Reiter – PERCEVAL
15
Stimulus material
30min 55sec long
HD resolution (1080p)
H.264/AVC compression – Quantization parameter11 versions, JND
Conversion to 16 bit YUV 4:2:0
Original audio track
Ulrich Reiter – PERCEVAL
16
Participants22 subjects (8 female, 14 male) in the age of 24-61 years
Test procedure and duration
Experimental procedure
Ulrich Reiter – PERCEVAL
Quality degradation initiated every 3 minsOne JND drop every 10 sec
17
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
18
Do the quality expectations decrease over time and with increased involvement in the content?
How fast can participants notice quality changes and at which quality level does this happen?
Is the quality level at which the change is noticed similar to the quality level set by the participant?
Questions to be answered
Ulrich Reiter – PERCEVAL
19
Three subsets of data (20x10 each) created:
1) average quality level of the last minute of each 3min time slot
2) response time to the automatic quality degradationright after the start of each 3 min time slot
3) quality level at the time when a user reacted to quality change
Data preparation
Ulrich Reiter – PERCEVAL
20
Results for subset data 1)
Main effects plot for quality levels averaged over last minute of each 3 min time section.
Ulrich Reiter – PERCEVAL
21
Main effects plot for reaction time to quality changes.
Results for subset data 2)
Ulrich Reiter – PERCEVAL
22
Main effects plot for quality levels at time when a userreacted to quality change
Results for subset data 3)
Ulrich Reiter – PERCEVAL
23
Results cont.
Comparison of sensitivity to quality changes underdifferent conditions.
Ulrich Reiter – PERCEVAL
QLRT: Quality Level at Reaction Time
AQL: Absolute Quality Level
24
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
25
Conclusions
Time dimension does not influence participants’ expectations with respect to the perceived quality. The reaction time to quality changes is relatively constantacross all time intervals.Participants are less sensitive to quality changes when the process is controlled externally than when they are in charge of the quality adjustment.
Ulrich Reiter – PERCEVAL
26
Outline
• PERCEVAL overview• Motivation (long duration)• Method description (long duration)• Experiment (long duration)• Results (long duration)• Conclusions• Food for thought
Ulrich Reiter – PERCEVAL
27
Food for thought• Content dependency
– Ideal bandwidth trade-off for audio vs. video quality depends on content type
• Impact of emotional state, fatigue– On perceived quality, on quality requirements / expectations– Use of EEG and body sensors
• Data analysis of non-traditional data– E.g. quantitative and qualitative, combined– Multivariate statistical methods
Ulrich Reiter – PERCEVAL
29
References
[1] http://perceval.no[2] ITU-R Recommendation BT.500-7: Methodology for the Subjective Assessment of the Quality of Television Pictures. International Telecommunication Union, Geneva (1996)[3] N. Staelens, S. Moens, W. Van den Broeck, I. Marien, B. Vermuelen, P. Lambert, R. Van de Walle and P. Demeester, “Assessing Quality of Experience of IPTV and Video on Demand Services in Real-Life Environments”, IEEE Transactions on Broadcasting, Vol.56, No.4, Dec. 2010.[4] Borowiak, A., Reiter, U., Svensson, U.P.: Quality Evaluation of Long Duration Audiovisual Content. In: The 9th Annual IEEE Consumer Communications and Networking Conference –Special Session on Quality of Experience (QoE) for Multimedia Communications, pp. 353--357, Las Vegas (2012)[5] Yang, X., Tan, Y., Ling, N.: Rate control for H.264 with two-step quantization parameter determination but single-pass encoding. In: EURASIP Journal on Applied Signal Processing, pp. 1--13, (2006)[6] Wang, H., Qian, X., Liu, G.: Inter Mode Decision Based on Just Noticeable Difference Profile. In: Proceedings of 2010 IEEE 17th International Conference on Image Processing, Hong Kong (2010)Thanks to Adam Borowiak and Ragnhild Eg for providing figures and slides!
Ulrich Reiter – PERCEVAL
