Upload
linda-barton
View
216
Download
0
Embed Size (px)
Citation preview
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Assessing Audio Visual Quality
P905 - AQUAVIT
Assessment of Quality for audio-visual signals over Internet and UMTS
Geoff Morrison, BT Labs
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Contents
• Why we are doing this
• What we want to achieve
• How we are going about it
• Who is doing it– BT, CSELT, Berkom
• When are we doing it– February 1999 to December 2000
• The Results
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
• Coding algorithms, such as MPEG-4 and H.263, can compress audio and video signals to bit rates<1 Mbit/s as available over mobile and IP networks.
• The impact of impairments, from compression and transmission, on users' opinion must be understood.
• Audiovisual communication must be evaluated under representative conditions and real tasks, such as conversational services and AV database retrieval.
• Network performance must be monitored to guarantee a minimum QoS. Need single-ended models to check audio and video transmission quality in real time.
Rationale
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Is the resulting application usable for purpose?
How best to assign a given total bit rate between video and audio?
Compression Impairments• Digital TV in studios is 216 Mbit/s
• Lossless compression typically only halves the bit rate
• To reach very low rates requires lossy audio and video compression, introducing significant distortions
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Transmission Impairments • AV bits <> data
• Audiovisual content has a “timeliness” aspect– Critical in conversational applications– Important in database retrieval applications
• Error detection and retransmission protocols not ideal for audiovisual content
– variable delay– variable throughput
• Compressed AV is very sensitive to transmission errors, packet loss
– more compression more sensitivity
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Error Handling• Error resilience tools for encoder
– structure compressed data so that damage is localised– optimise encoder options for the network performance
• Forward Error Correction– improve apparent network performance seen by decoder
• Error concealment at decoder– reconstruct approximations for damaged parts of pictures
and sound
• Combinations of above
How well do these work in practice?
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Project Objectives• Understand the potential of audio, video and AV
communication at bit rates < 1Mbit/s for UMTS and IP.
• Develop tools and techniques for simulation of complete audiovisual transmission systems.
• Develop quality evaluation methods for complete audiovisual transmission systems.
• Investigate the relationship between system parameters (network and terminal) and subjective quality.
• Provide guidelines for the system parameters to be used for new applications.
• Encourage the use of audiovisual communication on these dominant networks of the future
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Work Items
• Define the tests to be performed
• Develop/implement a UMTS test bed
• Develop/implement an IP test-bed
• Conduct subjective tests
• Make objective measurements and relate them to subjective results
• Disseminate the findings
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Definition of tests
• Literature survey for available test methods
• Select appropriate methods
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
UMTS Test Bed• Video codec (MPEG-4 and/or H.263)
• Audio codec (MPEG AAC, scalable and non-scalable)
• Error resilience tools (MPEG-4)
• MPEG-4 multiplex
• Transmission errors - simulated and perhaps actual
• Error recovery
• Demux
• Optimisation of parameters
• Tools to make objective measurements
• Production of material for subjective and objective tests
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
UMTS Test Bed
• Baseline non real-time test bed– MPEG-4 baseline video codec, AAC and G.723.1 audio, AL
layer and MPEG DMIF
• Enhanced non real-time test bed – Error resilience tools, layered audio
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
UMTS Test Bed
Server Application Audio decoder Video decoder
Network interface Network interface
Network
Server Client
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
IP Test Bed• Video codecs (H.263 and/or MPEG-4)
• Audio codecs (G.723.1, MPEG AAC scalable and n/s)
• Error resilience tools (ITU-T, MPEG, IETF)
• Packetisation
• Transmission errors - simulated, measured profiles and actual
• Error recovery
• Optimisation of parameters
• Tools to make objective data measurements
• Production of material for subjective and objective tests
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
IP Test Bed
• Baseline non real-time test bed– G.723.1, H.263, simplified RTP, simple network impairment
simulator
• Enhanced non real-time test bed– more codecs including MPEG, error resilience, complete
RTP, error concealment, enhanced network simulator
• Enhanced real-time test bed– real time image capture, encoding and decoding
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
IP Test Bed
Encoder Application
Audio Encoder
Server Application
Video Encoder
Network Interface
Network
Decoder Application
Audio Decoder Video Decoder
Network Interface
Encoder Decoder
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Subjective tests
• “Conventional” subjective tests covering a suitably wide range of variables
– the codecs– bit rate, video frame rate, audio bandwidth– Error conditions (packet loss, delay variability, …)– Protocols (CBR, VBR, ABR, RSVP, ..)– Application (one-way, conversational)
• Users may accept lower quality in return for mobility. Testing will seek an estimation of the threshold of acceptability of audio and video quality in mobile applications
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Objective measurements
• Objective assessment of audio quality– TOSQA, PAMS, ...
• Objective assessment of video quality– model of human vision– other algorithms to objectively estimate user’s opinion
• Compare subjective and objective assessments
• Investigate relationship between parameter values and subjective audio, video and AV quality assessments
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Double Ended Model
• Measuring instrument needs both original input to encoder and delivered version at decoder output
SystemunderTest
Meter
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Single Ended Model
• Measuring instrument needs only decoder output– more suitable for live performance monitoring
SystemunderTest
Meter
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Project Resources
20 man-monthsProjectManagementDissemination ofResultsDefinition ofTestsUMTS Testbed
IP Testbed
Subjective Tests
Objective Tests
AIMS’99 Workshop
Heidelberg, 11-12 May 1999
Results• Reports due in Q4 of 2000.
• The findings should be of benefit to at least 3 groups:– Network Providers. Results such as minimum desirable bit
rates and error characteristics should influence the planning and operation of IP and UMTS networks intended to carry audiovisual services.
– Application Providers. The results will provide reliable guidance about which applications and operating parameters will satisfactorily meet customers' expectations and those which will not.
– Researchers in the field of "human factors". It is expected that the project will advance the state of the art in perceptual modelling and in the understanding of cross-modal effects.