doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 1 IEEE TGT Streaming Media Apps Notice: This document has been prepared to assist IEEE It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein. Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEEs name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEEs sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE Working Group. If you have questions, contact the IEEE Patent Committee Administrator at.http:// Date: Authors: doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 2 Introduction From a user perspective, the performance of data applications can be clearly represented in terms of primary metrics (e.g. measured throughput, rate etc.) over various ranges. Performance of streaming media applications is more complex e.g. throughput alone is not a sufficient measure. The performance of streaming media apps may be expressed in terms of: Instantaneous, min, max, average throughput; forwarding rate, etc. Packet loss, errors, retransmissions, etc. Buffer overflow, average queue length, queue service rate, etc. Available bandwidth for background data traffic Audio/video quality Range Bit Rate, Media Content Some combination of the above All of the above doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 3 Performance of Streaming Media Apps Directly Observe User Perceived Video Quality Measure Secondary Metrics (Throughput, Packet loss, etc.) Assign Numerical Value to Observations Perform Correlation to User Experience Output: Performance of Streaming Media Apps Choose Streaming Media Content and Test Setup A combination of measurements may be considered Models for Correlation doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 4 Agenda We will discuss some measurement methodologies to show the impact of various metrics on the performance of streaming media apps. Measurement Methodology and Implications Single AP, Single STA Instantaneous throughput Packet loss or buffer overruns over range Average throughput over range User perceived video quality over range Single Access Point, Multiple STAs Available bandwidth for background traffic User perceived video quality QoS Conclusion doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 5 Measurement Methodology: Single AP, Single STA Access Points: D-Link DWL7000 AP Ch 52, Ch 11 D-Link DWL7100 AP Ch 52, Ch 11 Host: Dell Latitude D600 Client: Dell Latitude D600 Video quality in terms of throughput Measure instantaneous throughput Measure throughput (average, minimum and maximum) Perform correlation to perceived video quality actual video quality not measured Client device is within close range of AP range variation is not considered doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 6 Impact of Throughput on Streaming Media Apps D-Link DWL7000 AP a Average:22.3 Mbps Maximum:27.0 Mbps Minimum: 0.2 Mbps D-Link DWL7100 AP a Average:26.5 Mbps Maximum:29.5 Mbps Minimum: 0.5 Mbps RESULT: Second AP has better throughput performance. But we cannot be sure about the performance of streaming media content. For a SDTV Video stream of 6Mbps, the first AP might have better performance due to smaller dips in instantaneous throughput. SDTV HDTV Dropped Frames HDTV SDTV doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 7 Measurement Methodology: Single AP, Single STA Access Points: D-Link DWL7000 AP Ch 52, Ch 11 D-Link DWL7100 AP Ch 52, Ch 11 Host: Dell Latitude D600 Client: Dell Latitude D600 Video quality in terms of packet loss or buffer overruns over range Generate TCP stream with a target bit rate of 6 Mbps (SDTV) Count the number of re-transmission or buffer overruns (1 sec buffer) Perform correlation to perceived video quality actual video quality not observed doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 8 Impact of Packet Loss on Streaming Media RESULT: Performance of Card 2 is better than Card 1 because of: Lower percentage of dropped frames (evaluated by counting the number of TCP re-transmission) Better penetration through walls better range Note: We need to cover the scenario of real-time streaming (no buffering) Better Worse doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 9 Measurement Methodology: Single AP, Single Client Video quality in terms of throughput over range Measure average throughput for two different video clients Repeat measurements at various location in indoor environment Video Server Access Point Network Switch QCheck Server QCheck Client Video Client doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 10 Available Throughput at Various Locations RESULT: Client 1 has better data throughput for various locations. Client 1 should be able to provide better performance for streaming media applications. HDTV doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 11 Measurement Methodology: Single AP, Single Client Video quality in terms of throughput over range Stream video and note perceived video quality at each video client Repeat measurements at various location in indoor environment Add background data and note user perceived video quality Video Server Access Point Network Switch QCheck Server QCheck Client Video Client doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 12 User Perceived Video Quality RESULT: Both clients 1 and 2 have perfect video quality without any background traffic. Client 1 has better performance in the presence of background data traffic. User Perceived Video Quality Guide: 5 Perfect video quality, Perfect audio synchronization 4 Slight visible blur/blockiness, Perfect audio synchronization 3 Jerkiness or blockiness, Perfect audio synchronization 2 Greater jerkiness, Perfect audio synchronization 1 Greater jerkiness, out of sync audio 0 Unable to display video Unacceptable Acceptable Off-the-air HDTV content encoded in dvr-ms format doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 13 Measurement Methodology: Single AP, Multiple STAs Video Server Access Point Network Switch Chariot Server Chariot Client Video Client 1 Video Client 2 Video Client 3 Video quality in terms of available bandwidth for background data Measure throughput with and without actual video streams (6 Mbps) Stream multiple video streams with and without QoS prioritization Note the impact on perceived video quality All clients are within close range of AP range variation is not considered doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 14 Impact of Background Data on Streaming Media Apps RESULT: The perceived quality of video is only acceptable when QoS is enabled Note: There is available bandwidth even when QoS is disabled, but the perceived video quality is unacceptable. Unacceptable video performance Background Average Throughput doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 15 Impact of Background Data on Streaming Media Apps RESULT: The perceived quality of video is acceptable for both EDCA and HCCA QoS. Background Average Throughput doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 16 Streaming Media Applications Issues The goal for TGT is to determine the performance metrics that capture the impact of these issues: User Perceived Performance: The level of user perceived performance required to say that a certain media stream can run successfully. Throughput and Packet Loss Single/multiple media streams can run if certain throughput or packet loss thresholds are met Range Single/multiple media streams can run within x meters. Without background traffic With background traffic Impact of QoS Prioritization Single/multiple media streams can run with background traffic QoS enabled versus QoS disabled EDCA versus HCCA which mechanism provides better bandwidth utilization doc.: IEEE /0178r0 Submission March 2005 Fahd Pirzada - DellSlide 17 Conclusions The streaming media applications usage case is unique from data applications usage cases. We need to quantify the end-user experience using primary and secondary metrics. The choice of primary and secondary metrics should tackle the various issues unique to streaming media applications. In some instances the primary metrics may be measured directly. In other instances, the Primary metrics may not be measured directly. Secondary metrics may be measured Primary metrics may be derived using correlation and models There is a strong correlation between streaming media performance evaluated in a wide variety of controlled indoor environments. We need further correlation between different environments.