Control Theory Approaches to Resource …rts.eit.uni-kl.de/fileadmin/publication_files/Coelho...Control Theory Approaches to Resource Adaptation in Real Time Embedded Systems Master

Control Theory Approaches toResource Adaptation in Real Time

Embedded Systems

Master Thesis

Rodrigo Ferreira Coelho

Department of Electrical and Computer Engineering

Chair of Real Time Systems

Advisor:

Prof. Dipl.-Ing. Dr. Gerhard Fohler

May 20, 2008

Declaration

Hereby, I declare that I have done this thesis by myself without anyother help except the documents listed in the reference list.

Kaiserslautern, May 20, 2008

Rodrigo Ferreira Coelho

Abstract

Resources availability are strictly constrained on embedded systems. For instance,they have lesser computational power and memory availability than desktops /laptops machines, although they execute similar tasks. Therefore, one of the mostimportant issues in embedded systems is to find the optimal tradeoff between highperformance and low resource requirements. In this thesis we will propose theuse of control theory approaches to control the usage of constrained resources inembedded systems, thus improving the Quality of Service of applications runningin these systems. As case study, we will use the video adaptation of MPEG-2streams transmitted over bandwidth constrained wireless local area network. Thisis a challenging scenario, since both resource demand and resource constrainedvary continuously. The results presented throughout this thesis will show thatthe controlled system leads to significant improvement on the usage of availableresource, thus increasing the number of frames present on the adapted stream.

I

Acknowledgements

There would have been no better time for me to write the acknowledgments section:it is a sunny evening in the university (last one before my deadline), the spring hasjust begun and the university campus is quiet. It is a good moment to rememberthe people that were by my side - even if not physically - through this journey.

At first I would like to thank my parents who always offered me a peaceful andvery pleasant home during all my life. They have been motivating my search forknowledge and showing me the importance of a good education. Special thanksalso to my brother Renato, a great friend that (maybe) unintentionally has beenpushing me to achieve my goals, as during his life he has been following similarcareer as mine. I would like to specially thank my girlfriend for supporting me inthe hard and stressful moments of the Masters course. I am also grateful to myfriends in Brazil that encouraged me to join this Masters course abroad.

I am especially grateful to Prof. Gerhard Fohler for his guidance during this workand for his ability to keep such a nice atmosphere among the whole group thatworks in this chair. I also would like to thank the PhD students and my friendsthat work in this group: Alex Neundorf, Raphael Guerra and Ramon Serna fortheir wise reviews. Special thanks to Anand Kotra that helped me a lot duringdebugging phases of my thesis. I would like to thank Michal Izak, the PhD studentfrom the chair of Control Systems for his comments on my thesis.

The last, but not the least thanks, goes to God: “Thank God I am done!”

II

Contents

1. Motivation 1

2. Background 32.1. MPEG-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.1. Characteristics exploited for video compression . . . . . . . 42.1.2. Spatial and temporal redundancy . . . . . . . . . . . . . . . 52.1.3. Group of pictures and frame types . . . . . . . . . . . . . . 82.1.4. Brief introduction to MPEG-4 . . . . . . . . . . . . . . . . . 102.1.5. MPEG bitstreams . . . . . . . . . . . . . . . . . . . . . . . . 11

2.2. Real-Time Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.3. Control Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.3.1. Continuous control . . . . . . . . . . . . . . . . . . . . . . . 162.3.2. Discrete control . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.4. Wireless Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.4.1. Medium access control . . . . . . . . . . . . . . . . . . . . . 222.4.2. Bandwidth availability fluctuation . . . . . . . . . . . . . . . 23

3. Problem Formulation and Existing Solutions 243.1. Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.1.1. Video transmission overview . . . . . . . . . . . . . . . . . . 243.2. Existing Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.2.1. Reencode the video content . . . . . . . . . . . . . . . . . . 273.2.2. Reduce the frame rate . . . . . . . . . . . . . . . . . . . . . 27

3.3. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4. Proposed Solution 334.1. One-to-one Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344.2. Revisited Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.2.1. Augmented model . . . . . . . . . . . . . . . . . . . . . . . 384.3. The Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.3.1. Detailed system description . . . . . . . . . . . . . . . . . . 434.3.2. Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . 454.3.3. Controller design . . . . . . . . . . . . . . . . . . . . . . . . 46

III

Contents IV

4.3.4. Simulation and implementation issues . . . . . . . . . . . . . 504.4. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

5. Results 615.1. Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615.2. Incoming video streams . . . . . . . . . . . . . . . . . . . . . . . . . 615.3. Results analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625.4. Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

6. Conclusion and Future Work 84

A. Appendix 86A.1. PSNR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86A.2. MPEG-2 Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88A.3. Simulink Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

List of Figures

2.1. Spatial and temporal redundancy . . . . . . . . . . . . . . . . . . . 62.2. Spatial compression of residual images . . . . . . . . . . . . . . . . 62.3. Motion vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.4. Computation of residual images . . . . . . . . . . . . . . . . . . . . 72.5. Slice, macroblock, pixel . . . . . . . . . . . . . . . . . . . . . . . . . 82.6. Macroblock structure . . . . . . . . . . . . . . . . . . . . . . . . . . 92.7. Group of pictures (GOP) . . . . . . . . . . . . . . . . . . . . . . . . 92.8. Frames decoding dependencies . . . . . . . . . . . . . . . . . . . . . 102.9. Display and transmission order of frames . . . . . . . . . . . . . . . 112.10. Transport stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.11. Transport streams packet . . . . . . . . . . . . . . . . . . . . . . . . 132.12. Transport streams headers . . . . . . . . . . . . . . . . . . . . . . . 132.13. Program specific information tables . . . . . . . . . . . . . . . . . . 142.14. Deadline for video stream adaptation . . . . . . . . . . . . . . . . . 162.15. Open-loop control . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.16. Closed-loop control . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.17. Unit step response . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.18. Wireless network with 3 nodes . . . . . . . . . . . . . . . . . . . . . 23

3.1. Digital video transmission overview . . . . . . . . . . . . . . . . . . 263.2. QAFS selection loop . . . . . . . . . . . . . . . . . . . . . . . . . . 303.3. QAFS: available network bandwidth suboptimal utilization . . . . . 32

4.1. Video adaptation mapped into control loop diagram . . . . . . . . . 354.2. QAFS model used in simulation . . . . . . . . . . . . . . . . . . . . 364.3. Augumented video adaptation mapped into control loop diagram . 394.4. FIFO buffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.5. Buffer model with reset trigger for negative values . . . . . . . . . . 424.6. Signal tracking control for buffer bandwidth BBW . . . . . . . . . . 474.7. Simulink model: system overview . . . . . . . . . . . . . . . . . . . 514.8. Coefficients setup according to Ziegler-Nichols . . . . . . . . . . . . 524.9. System’s response to input step . . . . . . . . . . . . . . . . . . . . 534.10. Comparing the step response for two sets of coefficients . . . . . . . 54

V

List of Figures VI

4.11. Step response for realistic QAFS behavior . . . . . . . . . . . . . . 544.12. PID model used in the simulation . . . . . . . . . . . . . . . . . . . 554.13. Antiwindup connected to the PID controller . . . . . . . . . . . . . 554.14. Controller signals with windup . . . . . . . . . . . . . . . . . . . . . 574.15. Controller signals with antiwindup algorithm . . . . . . . . . . . . . 574.16. Network bandwidth usage improvement using the buffer controller . 60

5.1. Performance comparison for variety program, ANB profile var1 . . 645.2. Performance comparison for variety program, ANB profile var2 . . 655.3. Performance comparison for variety program, ANB profile var3 . . 665.4. Performance comparison for variety program, ANB profile var4 . . 675.5. Performance comparison for variety program, ANB profile var5 . . 685.6. Performance comparison for volleyball program, ANB profile voll1 . 695.7. Performance comparison for volleyball program, ANB profile voll2 . 705.8. Performance comparison for volleyball program, ANB profile voll3 . 715.9. Performance comparison for volleyball program, ANB profile voll4 . 725.10. Performance comparison for volleyball program, ANB profile voll5 . 735.11. Performance comparison for news program, ANB profile news1 . . 745.12. Performance comparison for news program, ANB profile news2 . . 755.13. Performance comparison for news program, ANB profile news3 . . 765.14. Performance comparison for news program, ANB profile news4 . . 775.15. Performance comparison for news program, ANB profile news5 . . 785.16. Performance comparison for cartoon program, ANB profile cart1 . 795.17. Performance comparison for cartoon program, ANB profile cart2 . 805.18. Performance comparison for cartoon program, ANB profile cart3 . 815.19. Performance comparison for cartoon program, ANB profile cart4 . 825.20. Performance comparison for cartoon program, ANB profile cart5 . 83

A.1. PSNR computation: comparison frame by frame. . . . . . . . . . . 87A.2. System overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89A.3. QAFS frame adaptation model . . . . . . . . . . . . . . . . . . . . 89A.4. The controller connection . . . . . . . . . . . . . . . . . . . . . . . 90A.5. PID algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90A.6. Antiwindup model . . . . . . . . . . . . . . . . . . . . . . . . . . . 90A.7. The buffer model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91A.8. Sensor overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91A.9. Buffer level sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

1. Motivation

Multimedia contents have become more present in our lives in the last decades.For many years analog video broadcasting is being vastly used. Its history datesfrom 1920s when the first tests on video transmission were done. But one majorproblem in analog signals is that they are very prone to noises. Although filterscan be used to reduce/cancel noises, in most of the cases, they can degrade theoriginal signal, and the result is very poor.

Attempting to solve problems with noisy signals, researchers started the first testswith digital video in the 1960s. Since then, much progress was done on digitalvideo signals, and nowadays the whole video broadcast process can be digitallyimplemented: starting from the video generated from a digital camera until thedigital monitor, passing through digital signal transceivers, recorders and play-ers. The introduction of digital video presented a new paradigm on multimediabroadcasting.

A wide variety of tools has been developed to be applied into digital multimediasignals. Their main target is to offer better quality services to the users, and tooptimize the usage of resources needed to broadcast, store and edit digital videosignals. To achieve these targets novel techniques have been developed, e.g., newdigital filters, image reconstruction, segmentation and selection algorithms; newtechniques for video compression; new algorithms for multiplexing etc. Practicalresults from the techniques mentioned above are images with noise reduction andpattern identification, use of fewer resources to store and transmit digital multi-media data and the possibility to share one resource with many signal sources.

Because increasing resource availability is directly related with increasing costs,enormous effort is being done to use resources in the most optimized manner.To achieve this goal, reducing the amount of redundant data to be stored/trans-mitted, and sharing the same resource among more applications are two commontechniques.

Moreover, improvements achieved by the semiconductors industry permitted dig-ital video processing tasks to move from desktop machines into small embeddedsystems. Embedded systems are designed according to their specific application,

1

1. Motivation 2

and furthermore, many of them are designed to be implemented in portable de-vices. Therefore it is a common practice does not over estimate the application’srequirements while designing these systems. Hence embedded systems face moreconstrained resources than a system based in a desktop machine. For instance,embedded systems offer much lesser computational power and memory availabilitythan a desktop system (considering both are designed on the same year). As previ-ously mentioned, many embedded systems are implemented into portable devices,thus power is mostly supplied by battery and energy consumption must be reducedas much as possible.

Methods used to adapt applications with constrained resources are roughly di-vided into inter- and intra-task adaptation. The former arranges the resourceavailabilities among a task set so that all tasks receive a fair portion of the avail-able resource [12], [8], [3]. The second one, adapts the task itself so that it canaccomplish its goals using the resources available to it [9], [8] . Researches havebeen done using control theory approaches to tackle the first problem for localand distributed systems [12]. In this thesis, we tackled the second problem us-ing a feedback control loop to adjust the resource availability perceived by theconsuming task. To explain this approach, transmission of MPEG-2 video streamover wireless network will be used as scenario, where both resources demands andavailability will vary continuously. The goal of this thesis is to develop a systemin which the usage of the available resources, for a real time application, will beoptimized.

The sequence of this thesis is organized as follows: Chapter 2 presents the the-oretical background of the concepts used in this thesis, Chapter 3 discusses theproblem formulation and presents the existing solutions, Chapter 4 we present ourapproach to handle the resource adaptation problem. In Chapter 5 we presentthe results achieved and in Chapter 6 we present the thesis conclusion and discussfuture works.

2. Background

2.1. MPEG-2

As previously mentioned, one method used to spare resources in a digital mul-timedia broadcasting system is to reduce the amount of data used to representinformation, in simpler words: data compression. As long as we are dealing withvideo broadcasting and/or video storage, specifying the multimedia stream signalis another issue that must be handled. To have a standard that specifies the audioand video compression and transmission, the International Standards Organiza-tion (ISO) founded the Moving Pictures Experts Group (MPEG). Although audiois part of the MPEG standard, we will keep the focus on video compression andtransmission since it is the most resource demanding part in most of all multimediacontents.

The first standard created by this group was MPEG-1, whose main target was tohave audio and video encoded into the bit rate of an audio Compact Disk: around1.5Mbit/s. For this reason, image resolution was 352 X 288 with frequency of 25pictures per second or 352 X 240 with frequency of 30 pictures per second. Whena higher transfer rate became available MPEG developed a new standard.

Then MPEG-2 was created providing more services. It became very popular sinceit was the standard chosen to be used in DVB (Digital Video Broadcast) and DVD(Digital Versatile Disk). A new paradigm presented by MPEG standards is thefact that they do not specify neither encoding nor decoding algorithms, rather theydo deeply specify the resultant bit stream used to store or transmit the multimediacontent. It permits many different algorithms for encoding and decoding (codecs)to be used and they are still MPEG-2 compliant as long as the resultant videostream follows the MPEG-2 Standard. In the sequel we present the MPEG-2standard. We start showing a few compression techniques and then the concept ofGroup of Pictures, types of frames, macro blocks and bit stream characteristics.

3

2. Background 4

2.1.1. Characteristics exploited for video compression

A good compression technique should deliver a result that compromise as little aspossible the quality perceived by the end user. In the case of video streaming thisconsumer is the human vision system. Therefore, understanding how we perceivemoving images is essential to efficiently compress video. The following list showsthe main characteristics of human vision system exploited by video compressiontechniques.

1. Flashing lights at frequencies higher than the so called critical flicker fre-quency (CCF) are perceived as continuous light. CCF is not constant though,it varies according to the image’s brightness. For example, considering thetypical TV set brightness CCF is around 50Hz . Not by coincidence this isthe frequency used to display the frames in PAL system. In summary, if asystem displays still images with a frequency higher than 50Hz we interpretthem as moving image. This characteristic is called persistence of vision.

2. The human being’s vision sensitivity to noise is much greater in images withlow contrast. Thus, after compression is applied, high contrast areas canafford more noise than low contrast ones .

3. The human’s vision sensors are called rods and cones. Rods are responsible todetect monochrome images, and they produce images with higher resolutionthan the ones formed by cones. Cones on the other hand, form color images.There are three different types of cones and each of them is sensible to lightwith different range of frequencies, i.e., each type responds to different colors. Roughly we can state that each type of cone responds either to red or greenor blue (R, G, B) forming the additive color matching

Video signal generation

Based on the additive color matching, the TV System was developed. All colorvideo cameras are composed of 3 individual simple cameras. Each of these cam-eras has a color filter (R, G, B) and the signal generated by them is related totheir respective filter. Reconstructing the value obtained from these “3 individualcameras” we get color video.

Monochrome video system, on the other hand, generates video from one singleindividual camera, from the so called luminance signal (Y ).

When color TV was introduced, compatibility with previous black and white

2. Background 5

(B/W) system was necessary. To achieve this compatibility, video was transmit-ted using the luminance and two color difference signals: Y , Cr = (R − Y ) andCb = (B−Y ). Since Y is formed as a function of R, G, B, ( Y = f(R,G,B) ), usingthis scheme, compatibility was possible with no need to transmit any extra signal:(B/W) TV sets used pure Y signal, and the color ones could reconstruct R, Gand B. Another reason to transmit luminance and chrominance signals separatelyis because the human’s vision system does not respond in the same way to colorand monochrome images (receptors are cones and rods, as described previously):differences in color are not resolved as well as in monochrome. Considering thischaracteristic, color difference signals can be sent with up to one quarter of the bitrate used for luminance and the human eye is unable to detect the degradation onthe image quality.

Analog video compression

Another characteristic that roots from analog TV systems is the interlaced video.To spare bandwidth in video transmission, the odd and even lines used to form oneimage (frame) are transmitted separately, this image with half horizontal resolutionis called field. CRT used in TV sets tends to blur images a bit, and for this reason,the lack of resolution is not detected in CRTs. If high resolution displays are usedto watch an interlaced video, a non “comfortable” effect will be detected on thescreen. To minimize it, most of video players offer a deinterlace function. Althoughpresent in the MPEG-2, usage of interlaced tends to be avoided in digital videos.

From this point, unless explicitly mentioned, no more analog video will be handled.

2.1.2. Spatial and temporal redundancy

One technique used by compression algorithms is to discard redundant data. Con-sidering video information, redundant data can be seen under 2 “planes”: spatialamong the same picture and temporal among subsequent pictures.

In the Figure 2.1(a), we can see a clear representation of spatial redundancy withina frame. This frame has the landscape as background and the upper part has thesame information: a bright, almost white sky. While compressing this frame, thereis no need to repeat the same information across the whole upper part of the frame,instead only reports that the information has to be repeated across it. Doing so,high level of compression can be achieved. Frames coded based only on its owncontents are called intra-coded frames (I-frames).

2. Background 6

(a) spatial redundancy (b) temporal redundancy

Figure 2.1.: Example of spatial and temporal redundancy

In Figure 2.1(b) temporal redundancy is depicted. We see that subsequent framesare very similar, and an efficient manner to represent them is to extract only thechanges presented in these frames across the time. They are the so called residualimages. Residual images are spatially compressed prior to transmission as shownin Figure 2.2.

Figure 2.2.: Spatial compression of residual images

Taking again Figure 2.1(b) as an example, let us analyze the car drifting fromthe left to the right side of the image. Since a large number of pixels changewithin consecutive frames, large residual images would be necessary to representthe background that was not there before and the car’s new position. MPEG-2allows encoding both the new background and the moving image. The way howto represent the new background will be explained latter. The car movement willbe represented by motion vectors. Encoders use motion estimators to generate themotion vectors. Some techniques used to estimate motions are block matching,gradient matching and phase correlation. However, the explanation about thesetechniques is out of the scope of this work though. Motion vectors can representhow a certain object has been displaced from a certain position in a frame to thenew position in the subsequent frame (in both horizontal and vertical directions).For instance, in Figure 2.3, the car’s license plate moves from left to right, so onlyhorizontal motion occurs.

2. Background 7

Figure 2.3.: Motion vectors inform the object’s origin in the previous frame

To reconstruct the new frame properly, it might not be enough only to reproducethe movement of an object, e.g., if the object moved from a bright to a dark area.In such a case where more information has to be transmitted, MPEG-2 standardallows using the so-called prediction error or residual image. Figure 2.4 depicts thisprocedure. To compute the residual value, the encoder uses the previous frame’smotion vectors and mimics the decoder’s actions as if only motion vectors wouldhave been sent. In this way it predicts locally the effect of motion vectors. After-wards, it compares the resultant frame with the original frame and the differenceis sent as residual. This residual is presented in Figure 2.4 as “prediction error”.

Figure 2.4.: Computation of residual images

So far we have not defined how big are the“objects”represented by motion vectors.Their size must be chosen to meet a compromise between decode performance andamount of data used to represent their motion. In MPEG-2 these objects are theso-called macroblocks: a 16x16 image area (Figure 2.5). Macroblocks have fixedboundaries in the frame, so it is not possible to move the macroblock itself. Instead,motion vectors inform in which pixel from the previous frame is the informationneeded to reconstruct the macroblock that it is being decoded.

2. Background 8

Figure 2.5.: Slice as a subset of a frame, macroblock as a subset of a slice and apixel as subset of a macroblock

Real objects in the video will not always match macroblocks perfectly. In this caseprediction error can be used to make up this lack of correctness.

MPEG-2 compresses also the way motion vectors are transmitted. The set of allhorizontally adjacent macroblocks is called slice (Figure 2.5). For each slice, themotion vector of its first macroblock represents the absolute motion value. In allother macroblocks within the slice, the motion vector represents a relative valueto the first one.

As presented in Section 2.1.1, chrominance information can be expressed withlower resolution than luminance, and human vision will not detect this difference.Macroblocks are formed exploiting this property. Most of MPEG-2 profiles usemacroblock with a 4:2:0 structure, i.e., for one macroblock with area of 16x16pixels, the sampling areas used to represent color difference have 8x8 pixels andthe luminance have 16x16 pixels (Figure 2.6(a)) Other profiles define macroblockstructures as 4:2:2, where luminance sampling areas have 16x16 pixels, and colordifference are defined by 2 sets of 8x8 pixels each (Figure 2.6(a)). Table A.1presentsall profiles and their specifications defined in the MPEG-2 standard.

2.1.3. Group of pictures and frame types

As mentioned in the previous section, a frame can be reconstructed taking in-formation from the previous one plus changes represented by motion vectors andprediction error. Frames constructed in this way are called P -frames (Predictive).Long videos composed by a large sequence of P -frames are prone to error propa-gation. For instance, if for any reason the root frame (I-frame) is missing, even ifthe subsequent P -frames are present, they can not be reconstructed.

MPEG-2 does not impose any fixed structure to compose a video, i.e., a sequence

2. Background 9

(a) Macroblock structure 4:2:0

(b) Macroblock structure 4:2:2

Figure 2.6.: Macroblock structures [16].

of either dozens of consecutives I-frames or P -frames can be used. However thoseboth extremes are wise choices. They would result in a stream with large data sizeor be very sensitive to error respectively.

An important concept defined by MPEG, and that will be very often used in thiswork, is the so-called Group of Pictures (GOP). It is defined as the set of framesbetween 2 consecutives I-frames, including the first one (Figure 2.7).

Figure 2.7.: GOP, group of pictures formed by all frames between two I-frames,including the first one [16]

P -frames can handle pretty well object movements using motion vectors, theyleave one unsolved issue. Considering an encoded P -framei, apart from the en-coded motion vector, this frame will reveal some background content that was not

2. Background 10

present in the previous framei−1. If we consider, for instance the car’s movementin Figure 2.1(b), this background will be present in the next framei+1. So as to beable to use both information from previous and next frame, MPEG-2 allows use ofB-frames (bi-directional). Again, considering the first frame of Figure 2.1(b), asframei. A new part of the landscape appears on the left side (background) as thecar drifts to the right on framei+1. If framei+1 is encoded as a B-frame, it does notneed to encode this new background. It simply stores the information reportingthat data necessary to encode the background will be present in the framei+2.

A common practice is to have 2 B-frames between 2 P -frames or between oneI-frame and one P -frame. For instance, IBBPBBPBBPBB is a typical GOPstructure for broadcasting purposes. Since the last frame is of type B, the nextI-frame is needed to decode it. This is called open GOP. A closed GOP can beobtained with this structure if all macroblocks of the last B-frame are forwardpredicted. Figure 2.8 displays the decoding dependencies within a GOP.

Figure 2.8.: B and P -Frames depend on other frames to be decoded: bi-directionaland previous

Use of B-frames created a new issue on decoders: they must know the subsequentframe prior to decode a B-frame. MPEG-2 solved this issue by rearranging thesequence at which frames are transmitted. Doing so, whenever the decoder receivesa B-frame, the 2 necessary frames to decode it will already be stored on a buffer.Figure 2.9 shows the difference between display and transmission order.

2.1.4. Brief introduction to MPEG-4

Before start with the introduction to the MPEG-4, we will answer one curiousquestion: where is MPEG-3 ? The MPEG-3 standard never existed as a finalstandard. The standard that would have become MPEG-3 was completed before

2. Background 11

Figure 2.9.: Display and transmission order of frames

the MPEG-2 had been published, thus MPEG-2 incorporated MPEG-3.

MPEG-4 has broken important paradigms present until MPEG-2. For instance,moving objects are not bounded by macroblocks: they can be encoded with ar-bitrary shapes. Moreover, background images can be coded independently frommoving objects in front of it. It is possible to declare one object in the imageto be two or tridimensional, opaque or translucent. For instance if one object is3D, the decoder can perform faster, considering some symmetric properties of thisobject. MPEG-4 presents new dedicated standards for face and body animation.Furthermore, MPEG-4 introduced the temporal scalability, where a base level bitstream can be augmented to produce an output with higher frame rate.

In this thesis, we do not make use of MPEG-4 video streams, hence no furtherdetails about MPEG-4 will be explained. Detailed information about MPEG-4can be found in [16].

2.1.5. MPEG bitstreams

After being encoded, the frames created need either to be locally stored or trans-mitted. Therefore, MPEG-2 standard defines two types of bit streams: programstream and transport stream. The former one is designed for use in an environ-ment almost error-free and can handle only one program. This is the type of bitstream used to store video in DVDs. On the other hand, transport stream canhandle multiprogram and includes artifacts to handle noisy environments. Sincethis thesis concerns transmission of MPEG-2 streams over the network, no furtherinformation about program stream will be presented. The structure used to buildthe transport stream is depicted in the Figure 2.10.

2. Background 12

Figure 2.10.: Transport stream constitution

Elementary Stream (ES) continuous bit stream with only one type of encodeddata: either video or audio. Pictures are not necessarily in the correct order.

Packetized Elementary Stream (PES) bit stream formed by packets of elemen-tary stream data. Each packet starts with a header containing the start code,followed by a code that identifies its type (audio, video) and a code that in-forms the packet length. Moreover, optional information can be includedinto this header i.e. presentation and decode time stamps (PTS and DTS)(DTS/PTS do not need to be present in every PES header, instead theycan be transmitted at a maximum interval of 100ms in transport streams).Decode time stamps inform the decoder in which order the frames must bedecoded. As described in the previous paragraphs, B-frames require the nextP - or I-frame to be decoded. This rearrangement is done considering PTSand DTS. See Figure2.9.

Transport Stream (TS) stream definition designed to carry multiplexed audioand video channels (although Single Program Transport Stream may beused). It is built of constant sizes packet: 188 bytes long. This charac-teristic makes it easier to develop error checking algorithms in the layersabove transport stream. If this is the case, the bits added by upper layerswill increase the original bit stream size. The diagram with transport streamheader is shown in Figure2.11. Each packet in the transport stream will re-ceive one Packet Id (PID), which is a 13 bit wide field that identifies the

2. Background 13

packet’s content. For performance reasons, the transport stream header isnormally short, although extra information may be included for some pack-ets. In those cases, payload is sacrificed. Figure 2.12 presents the transportstream header.

Figure 2.11.: Transport stream packets have fixed size: 188 bytes

Figure 2.12.: Presentation of transport stream headers

Program Specific Information (PSI)

Many programs can be multiplexed in the transport stream. To select which pro-gram will be decoded, the demultiplexer uses the Program Specific Information(PSI). This information is stored in two tables: Program Association Table (PAT)

2. Background 14

and Program Map Table (PMT). At regular interval of time, the PAT is transmit-ted in the transport stream. PATs contain the relation table that informs for eachprogram their respective PMT. PMTs are tables that relate one program with itsrelated streams. Each stream has its own PID. For instance, in Figure 2.13 thePMT 1 contains the information concerning the streams of Program 0: packetswith PID=51 contain one video stream of the Program 0, PID=53 contain oneaudio stream etc. Packets with PID=0 transmit the PAT, therefore, when thedecoder starts, it searches for the packet with PID=0 and stores the PAT. Then,when the user selects the channel, the demultiplexer can collect the correspondentpackets.

Figure 2.13.: Program specific information tables

2. Background 15

2.2. Real-Time Systems

The term real time is frequently misused. It is often stated that “real time systemsare very fast systems”. However, the actual target of a real time system is toprovide determinism: correctly react to the environment’s input, in time to achievea successful result. At the process level, [2] the main difference between a real timeand a non real time task is that the former one is characterized by a deadline, whichis the maximum time within which it must complete its execution.

Real time systems are classified into two major categories: hard and soft real timesystems. In the former one, the system is designed such that no one task will missits deadline. For the latest one, although the tasks must finish their executionwithin their deadline, if occasionally deadlines are missed, no serious consequencesfor the system occur. For example, a task that reacts to the emergency button ina factory production line is a hard real time task. If the task misses its deadline,not only financial lost will occur, but also workers can be injured. On the otherhand, as an example of soft real time task, we can cite the decoding task of adigital video satellite receiver: frames must be decoded within their deadline to sothat the video is correctly displayed. However, deadline misses will only incur ina drop of the perceived quality on the decoded video.

As we presented in Chapter 1, we will analyze the video adaptation of MPEG-2streams throughout this thesis, and will consider this adaptation as a soft real timetask. In figure (Figure 2.14) we present the time constraints that specify this task:

arrival time (or release time) ri: is the time at which the task becomes ready forexecution in the system

computation time ci: is the time needed for the task to be completed

deadline Di, di: the relative deadline di is the maximum time the system can useto complete the task, and the absolute deadline is given by: Di = ri + di

2.3. Control Theory

“Engineering is concerned with the understanding and controllingthe materials and forces of nature for the benefit of humankind. Con-trol system engineers are concerned with understanding and controllingsegments of their environment, often called system, to provide usefuleconomic products for society” [4]

2. Background 16

Figure 2.14.: Deadline for video stream adaptation under constrained networkbandwidth

2.3.1. Continuous control

The goal of a controlled system is it to drive the system’s output to a desiredvalue, according to input signal given to the system. Figure 2.15 presents thenaıve approach to reach this goal. There, an actuator is designed such that itadjusts the reference input (r) into the signal applied to the process (u). It is thenexpected that the system’s output (y)will correspond to the reference input. Onthis scenario, two major problems are not taken into consideration: model errorsand presence of disturbances.

Figure 2.15.: Open-loop control

Figure 2.16.: Closed-loop control

To design a correct controller, it is necessary to know how the process behaves.Therefore, a process model needs to be built. A good model must be able to repro-duce the real process behavior. However, complex systems might be very difficult

2. Background 17

or impossible to be perfectly modeled. Therefore simplified models are generatedto reduced computation efforts, or even because a complete understanding of thewhole system behavior is not feasible. Moreover, the system is exposed to dis-turbances that influence the output in a manner hard to predict, i.e., a conveyorbelt might endure higher friction after many operating hours and therefore cannot achieve the predicted speed with the given input signal. To overcome thesequasi-unpredictable changes, it is a common technique to use a feedback loop, asshown in Figure 2.16.

Both [4] and [5] explain in details the effects of feedback in a control system.Main disadvantages on the use of feedback are: the reduction on the total loopgain and the possibility to lead a stable system to instability. The latter onecan be compensated by increasing the controller gain. The former one is of greatimportance and must be investigated. On the other hand, the advantages achievedby a feedback loop motivate its vast use: feedback reduces the system’s sensitivityto disturbances, permits the designer to enforce the system’s dynamic responseto meet specific performance criteria, and even to stabilize an unstable system.Figure 2.17 depicts the main performance parameters in time-domain:

0 5 10 15 20 250

0.2

0.4

0.6

0.8

1

1.2

1.4

Mp

Tp TsTr

Figure 2.17.: Unit step response

Rise Time tr: The time the system takes to reach the vicinity of its new set point

2. Background 18

(in Figure 2.17 measured between 10% to 90% of the final value).

Settling Time ts: The time it takes the system transient to decay (in Figure 2.17defined as 98% of the final value).

Overshoot Mp: Percentage value that expresses the maximum value of a systemresponse in relation to its final value.

Peak Time tp: The time it takes the system to reach the maximum overshootpoint.

One method used to express the dynamic response of a linear system is using differ-ential equations. Furthermore, in linear systems the superposition property holds.Therefore the concept of convolution is used to analyze the system response toinput signals. Once the response to an impulse input (h(t)) is known, it is possibleto apply the convolution integral (Equation (2.1)) to any input signal, and obtainthe system response to this input. However, the convolution is a tedious operationto compute. It is possible to avoid the use of convolution operation by applyingthe Laplace transform (L ). The time-domain convolution operation is replacedby simple multiplication in frequency-domain. For instance, the result achievedby convolving the impulse response (h(τ)) with an input signal in time-domain,is equivalent to the multiplication of their Laplace transforms1. Furthermore, thedifferential equations that describe the system dynamic are replaced by algebraicequations.

y(t) =

∫∞

−∞

u(t− τ)h(τ) dτ (2.1)

Therefore, analysis in frequency-domain is widely used, and the system output infrequency domain (Y (s)) is represented by multiplication of the transfer functiontimes the Laplace transform of the input signal (Eq. (2.2))

Y (s) = H(s) × U(s) (2.2)

If the time analysis of the system’s output is required, the inverse Laplace transform(L −1)is applied to Y (s).

1The system’s response to an impulse input in time-domain (for initial conditions equal to zero)is equivalent to the system’s transfer function (H(s))

H(s) =

∫∞

−∞

h(τ) est dτ

[5]

2. Background 19

In addition, another important topic in control theory is the interpretation of thetransfer function’s denominator (characteristic equation’s) roots. They representhow the amplitude of the output signal decays in time domain. It can be proved ([4]and [5]) that these roots, called poles, are present as exponents of the exponentialcoefficients that form the time-domain output function. For instance, if

H(s) =1

s2 + 3s+ 2(2.3)

the poles ares = −1, s = −2

and H(s) can be expressed as

H(s) =1

s+ 1−

1

s+ 2

Applying the inverse Laplace transform, we get

y(t) = e−t − e−2t

Therefore, one can conclude that, for linear time invariant systems, if the real partof all poles have negative sign (and no more than one pole is equal to zero), thesystem is stable, i.e.

limt→∞

y(t) = 0

One method to classify a system is according to its steady-state error. For instance,if a system presents no steady-state error for a step impulse input, it is said thatthis system is of type 0. To compute this value, the final value theorem is used.This theorem states that, if the system is stable, then it holds:

limt→∞

y(t) = lims→0

sY (s) (2.4)

Table 2.1 presents the system types and their respective steady-state errors fordifferent input signals:

InputType Step Ramp Parabola

Type 0 Constant ∞ ∞Type 1 0 Constant ∞Type 2 0 0 Constant

Table 2.1.: Steady-state error as function of system type

2. Background 20

2.3.2. Discrete control

Continuous control is based on the fact that the system is all implemented inanalog electronic, thus electronic continuous signals flow across the whole system.For computer based systems (e.g., microcontrollers or microprocessors) all com-putations occur in discrete time. Moreover, unlike analog electronics, computerscannot integrate. Therefore, to compute differential equations, computers need toapproximate them to algebraic equations called difference equations These differ-ence equations are recurrent equations that, for linear and causal systems2, arerepresented as in Eq. (2.6). Suppose we call the input signals up to the kth samplee0, e1, . . . ek, and the output signals prior that time u0, u1, . . . uk−1:

uk = f(e0, . . . ek; u0, . . . uk−1) (2.5)

Assuming Eq. (2.5) is linear and depends on n elements of u, and on m elementsof e:

uk = −a1uk−1 − a2uk−2 − · · · − anuk−n

+b1ek−1 + b2ek−2 + · · ·+ bnek−m (2.6)

Let us analyze the difference equation Eq. (2.7). Assuming u(k) = Azk, we getEq. (2.8) and after some algebraic manipulations Eq. (2.9) [6].

uk = uk−1 + uk−2 (2.7)

Azk = Azk−1 + Azk−2 (2.8)

z2 = z + 1 (2.9)

In [6] it is proved that, if z1, z2 are the roots of Eq. (2.9), then:

uk = A1 zk1 + A2 z

k2 (2.10)

Moreover, it can be stated that if , e.g., z1 is greater than |1|, the term zk1 will

grow to infinity as k grows. Thus, if any root of Eq. (2.9) has a magnitude greaterthan 1 (outside a unitary circle), the system is said to be unstable.

For discrete time systems, the Z -Transform plays the same role in the differenceequation as the Laplace transform does in the differential equation. For a signalwith discrete values e0, e1, . . . ek, its Z -transform is given by:

E(z) =∞∑

k=−∞

ek z−k (2.11)

2Causal systems are those whose inputs do not depend on their outputs, e.g., all physicalsystems are causal.

2. Background 21

The discrete transfer function is presented in the Z -domain, and is defined as theratio of the transform of the output to the transform of the input, as presented inEq. (2.12).

H(z) =U(z)

E(z)(2.12)

Considering the difference equation presented in Eq. (2.6), a direct mapping be-tween the difference equation and the Z -transform is given by:

H(z) =b0 + b1 z

−1 + b2 z−2 + · · · + bm z

−m

1 + a1 z−1 + a2 z−2 + · · ·+ an z−n(2.13)

For example, the numeric forward Euler’s integration is given by the followingdifference equation [6]:

uk = uk−1 + Ts ek−1 (2.14)

Thus, the transfer function of an integrator implemented in a computer systemaccording to the forward Euler algorithm is given by:

H(z) =Ts

1 − z−1(2.15)

An important topic for digital sampled systems is the choice of the sampling period.This period determines when the digital part of the system must acquire data andexecute its computations According to the sampling theorem, a signal can berecovered from its samples if the sampling frequency is at least twice the highestfrequency of the signal [6].

2.4. Wireless Network

Use of wireless networks has been vastly increased on the last decade. The largenumber of wireless network related equipments on the market and their low cost,have made wireless networks popular. They replace and expand the previouslyestablished Ethernet LANs as they can offer more mobility to the users and savesall troubles caused by Ethernet cabling. However, wireless networks are proneto much more interferences than wired ones. For instance, noise caused by thereceiver circuitry can corrupt the received signal as well as interference from otherdevices operating on the same frequency like microwave ovens.

The IEEE 802.11 specification (ISO/IEC 8802-11) is an international standarddescribing the characteristics of a wireless local area network (WLAN). The 802.11standard specifies 2 layers of ISO OSI 7 layers model: the physical layer (PHY) and

2. Background 22

the medium access control layer (MAC). The former one specifies the transmissionand reception of 802.11 frames. The latest one provides a variety of functions thatsupport the operation of 802.11-based wireless LANs. In general, the MAC Layermanages and maintains communications between 802.11 stations (radio networkcards and access points) by coordinating access to a shared radio channel andutilizing protocols that improve communications over a wireless medium. In thissection we will give a brief explanation about the standard 802.11 medium accesstechniques that are relevant for the rest of this thesis.

2.4.1. Medium access control

Before transmitting frames, a station must first gain access to the medium, whichis a radio channel that stations share. The 802.11 standard defines two forms ofmedium access, distributed coordination function (DCF) and point coordinationfunction (PCF). DCF is based on the CSMA/CA (carrier sense multiple accesswith collision avoidance) protocol. With DCF, 802.11 stations compete for accessthe medium and attempt to send frames when no other station is transmitting.If another station is sending a frame, stations have to wait until the channel isfree. Before a station access the medium, the MAC Layer checks the value of itsnetwork allocation vector (NAV). NAV is a counter resident at each station thatrepresents the amount of time that the previous frame needs to send its frame.If the NAV is equal to zero, the station is enabled to try to send the next frame.Before transmitting a frame, a station calculates the amount of time necessary tosend the frame based on the frame’s length and data rate. The station places avalue representing this time in the duration field in the header of the frame. Thisprocess reserves the medium for the sending station. If the station tries to accessa busy medium, it starts a random back off timer. It then waits for a randomtime and retries to get access to the medium. The use of a random number toinitialize the timer, avoids that multiple stations try to start their transmissionsimultaneously and get blocked again.

A transmitting station cannot listen for collisions while sending data, mainly be-cause the station cannot have its receiver on while transmitting the frame. As aresult, the receiving station needs to send an acknowledgement (ACK) if it detectsno errors in the received frame. If the sending station does not receive an ACKafter a specified period of time, the sending station will assume that there was acollision (or RF interference) and retransmit the frame.

In PCF, on the other hand, no collision occurs since the transmission order iscontrolled by the access point. Stations cannot transmit frames unless the access

2. Background 23

point polls them first. The way how the access point grants the medium access tothe nodes is not defined by the standard, and can be implemented according toeach application.

2.4.2. Bandwidth availability fluctuation

Considering the medium access mechanisms and the non reliability of the wire-less medium, it is straightforward to realize that fluctuations on the bandwidthavailability of wireless networks are large and is frequently present. For instance,assume an ad-hoc configuration with 3 nodes A, B, C (Figure 2.18). Node B istransmitting to node C. Node A and C can listen to this signal and suspend theirfuture transmissions. After the transmission, C sends the ACK signal to B, but ifinterferences deny this acknowledge signal to reach B, the transmission from B toC restarts. During the whole process, A is not allowed to access the medium.

Figure 2.18.: Wireless network with 3 nodes

3. Problem Formulation andExisting Solutions

3.1. Problem Formulation

As described in Chapter 1 the goal of this thesis is to optimize the usage of avail-able resources in a real time environment by means of control theory. We usedtransmission of MPEG-2 video over wireless network as case study. The first stepto understand the cause and effects of this scenario is to have a model of thesystem we want to control. In this chapter a very simple model of MPEG-2 trans-mission over network will be depicted so as to present an overview scenario. Amore detailed model including the approach used to design the controller itself willbe presented in Chapter 4. Nevertheless, details concerning wireless network datatransmission are not scope of this work and will not be presented.

3.1.1. Video transmission overview

Digital video transmission over network has become a very common and importantapplication in the last decade. It has been used for purposes that range fromentertainment (e.g. watching sports, news, video-on-demand, home-made video) tobusiness and researching (e.g. video conference, video medicine). The interestingand challenging characteristic of such application is the fact that both resourcesavailability and resources demand varies continuously and independently from eachother.

Let us consider the transmission of a movie over wireless network. The demandresources will vary continuously imposed by the video contents. Detailed scenesand scenes with fast motion (e.g. action movies) will generate larger data, withlarge motion vectors and residual images that will require large amount of resourcesto transmit the video (as explained in Section 2.1.3). We can compare it with thevideo contents transmitted during the news, where the news anchor is shown for along time. In this case motion vectors represent only the anchor’s movement and

24

3. Problem Formulation and Existing Solutions 25

almost no residual images are necessary. This comparison exposes how big thedifference between the data used in video transmission can be.

On the other hand, the resource availability also varies continuously and inde-pendent from resource demands. It is a known phenomena that the bandwidthavailable to a wireless node face constant structural fluctuations (e.g. caused bya moving node, heavy cross traffic, RF interferences like the ones caused by mi-crowave oven etc). In these cases also the resource availability will suffer of largevariations.

We can draw a parallel scenario with a “common life” situation. Let us considerthe daily water consumption of a family house. Suppose the public water supplycompany supplies a water flow that is non constant, but enough to supply the“normal” family’s demand in most of the days. Also, the family does not have aconstant water demand. For instance, whenever the kids play soccer, they drinkmore water, take longer showers and therefore the water consumption increasessubstantially on those days. Assuming it is not possible to ask the water supplycompany to increase the water flow, it is family’s responsibility to deal with thisproblem: their demand varies drastically and the water availability varies as well.

Another implication takes place when we consider the “real” transmission overnetwork. As mentioned in Chapter 2.4 the collision detection polices in IEE 802.11standard will avoid data transmission if the node can not access the media anddata will not be sent. This data will be kept in a buffer to be sent later. In caseof an overloaded network, this situation will occur often. Analyzing this scenario,a straightforward conclusion is that if we try to overuse the available networkbandwidth, the sending buffer will accumulate more and more data. If we do notadapt the video stream to new network state, long delays will be generated on theclient side. This is situation must be avoided. In those cases is preferable to sendless data, reducing the bit rate either by decreasing the resolution or by decreasingthe frame rate.

Figure 3.1 depicts the basic scheme for digital video transmission. In our scenario,a video server is responsible to generate an adapted video stream Vout from anoriginal stream Vin and send it to a client over network. On the client side, aMPEG-2 decoder will decode the receiving video stream and play it.

Considering the architecture used in the video systems nowadays, the block pre-sented as “Stream Adapter” in Figure 3.1 is not implemented in a single hardware.Instead it is part of a set of software applications sharing the same hardware.For those architectures, one must take into account feasibility tests to ensure thevideo adaptation completeness. Considerations concerning the impact of videoadaptation application on a multitask hardware is presented in [9]. Moreover, as


Figure 3.1.: Digital video transmission overview

described in section 2.2, large delays in stream adaptations are not allowed. Thatleads us to conclude that besides the network availability constraint, another con-straint that must not be forgotten is the fact that we should implement the videoadaptation application under an architecture which has limited resources as well,i.e. we can not afford to implement a solution which will consume large amountof computational power.

3.2. Existing Solutions

To handle the problem of video stream adaptation under constrained and variableresources environment, many solutions have been studied so far. The idea is verystraightforward: if resources availability increases, the adaptation task is allowed toproduce a more resource-consuming result. Otherwise, when resource availability islow, it has to produce an output that fits on the new resource availability. Some ofthese techniques will be presented on the next section and their pros and cons willbe highlighted. For any of those approaches, we that assume a metric representingthe available network bandwidth value is available.

Network bandwidth estimation is by itself a wide and already extensively exploitedsubject. Many algorithms provide metrics that represent the network status withdifferent approaches, e.g., [13], [10], . For the sake of simplicity, we will assumethat we can read this metric from a given algorithm according to [11].


3.2.1. Reencode the video content

One possible technique to adapt the incoming video stream according to the avail-able network bandwidth is to decode this stream and reencode it again. In thisapproach the incoming video stream Vin is decoded (i.e. the frames are decom-pressed to form the raw input video). Afterwards an encoder is used to composethe output video stream Vout. Great pros on this method are, first the fact that,since we will reencode the stream, we can impose the encoder strict constraints(video resolution, bit rate, frame rate, select the encoding algorithm etc) so thata reencoded video will demand almost exactly the available network bandwidth.Second, we can establish a smooth video if not many frames are skipped in the reen-coded video. For instance we can switch between different MPEG-2 profiles/levelsso that the output stream maintains the same frame rate.

On the other hand, to execute this process on the fly, high computational powerwould be demanded. To achieve an acceptable performance so that the decoding-and-reencoding task meets its deadline (i.e. does not insert long delays), either adedicated, powerful system should be available to this purpose, or other tasks thatshare the same processor might not be schedulable. As mentioned previously, themarket tends to implement as many applications as possible on the same hard-ware, thus develop a new solution based on a dedicated processor is a prohibitiveapproach. However, the methods presented in this section are widely used whenthe scenario has a different configuration.

Scalable video

Assuming the video server does not receive the input video from a video stream,but from raw video source. In these cases, the server has to encode and constructthe output video stream from scratch, thus it can take into account the availableresources (e.g. network bandwidth, CPU etc) in the moment when it encodes andthen generate the stream accordingly.

In another possible scenario if the stream is to be decoded under constrainedresources, scalable video can be used in such a way that according to the resourcesavailability the incoming frames are decoded in different quality levels.

3.2.2. Reduce the frame rate

If we assume, that we should not adapt the incoming video stream by decodingand reencoding it on the fly, a good approach is to skip frames. In this case, we


will change neither the video resolution nor any other encoding parameter but theframe rate.

In this technique we assume that not all frames can be transmitted and some haveto be discarded. Based on a selection algorithm, frames that should not be presenton the output video stream are removed and the output stream is reconstructed.The selection algorithm will determine which frames are to be skipped so that thefinal output stream does not require resources beyond its availability.

The naıve approach to skip frames does not consider the impact of discarded frameson the final video quality, instead, it skips frames taking into account only theresource constraints. We agree that we should not abuse the available resources,but the “final product must satisfy the customer as much as possible”. Thus, it isimportant to discard frames in a manner that the output stream has a “as good aspossible” quality. Researches have proved that it is possible to reach better videoquality if the frame selection is made based on some frames characteristics. Aninteresting approach to adapt MPEG-2 video streams under constrained resources,called Quality Aware Frame Selection (QAFS), has been presented in [9] and itsmain characteristics will be presented following.

QAFS

The performance of QAFS does not depend, in any manner, on the codec used toencode the MPEG stream. In QAFS the video stream is divided into parts. Eachframe within this part of video is then ranked according to its importance. In thecase that the availability of resources is not big enough to send the whole inputstream, frames will be skipped according to the ranking given to them. A naturalchoice is to analyze the video stream GOP-by-GOP and rank the frames withinthis GOP. The criteria to set frame importance (high importance = high priority)within a GOP is briefly presented next. Considering the network bandwidth asthe constrained resource, the priorities are assigned as follows:

Frame Type : This criteria states that frames of type I must have the highestpriority in a GOP and the ones of type B have the lowest. As describedin section 2.1.3, if one I frame is not present, the whole GOP can not bedecoded. B-frames on the other hand are not used to decode any otherframe except itself. P -frames are located in between since their absence willcompromise some frames but not the whole GOP (see next item).

Frame Position in the GOP : As shown in Figure 2.8, a missing P-frame willcompromise all subsequent frames in a GOP as well as the two previous B-


frames. So, the closer to the GOP’s begin (i.e. I-frame inside the GOP) themore important is a P-frame. This criterion is applied only for P -frames.

Frame Size : This is a criterion to sort the B-frames. Skipping a B-frame doesnot avoid any other frame to be decoded, or simply: the impact of missingone B-frame is one frame less being decoded. Considering we want to savenetwork bandwidth, greater B-frames must be discarded first.

Skipping Distribution : This is another criterion to sort the B-frames. Skip-ping B-frames evenly produces a smother video than skipping subsequentB-frames. Let us make the following assumptions concerning the incomingvideo stream:

Frame rate = 20 FPS

⇒ Frame period =1

20= 0.05s

Let us analyze the GOP: IBBPBBPBBPBB and assume we need to dis-card 4 B-frames. If we discard 4 consecutive B-frames, e.g.:

B1, B2, B3, B4 (3.1)

the adapted GOP is:I −− P −−

︸︷︷︸

0.2s of 0.25s

PBBPBB. (3.2)

All skipped frames are removed from 0.25s of video stream. In this periodof time we will miss twice the contents of two frames (2 × 2 × 0.1 = 0.2s).The worst frame rate for this GOP is:

I −− P −−︸︷︷︸

0.25s

PBBPBB

FR =1 frame

0.25s⇒ 4 FPS.

Instead, if for instance the following frames are discarded:

B1, B3, B5, B7 (3.3)

the adapted GOP is:I −BP − BP − BP−

︸︷︷︸

0.2s of 0.5s

B. (3.4)

The time gap left by the skipped frames will be spread within 10 frames⇔ 0.5s, and the worst frame rate in this GOP will be:


I −B︸︷︷︸

0.1s

P − BP − BP −B

FR =1 frame

0.1s⇒ 10 FPS.

As described in Section 2.1.1, higher frame rates are interpreted as smothermovements. Thus, the perceived quality of the adapted GOP 3.4, will behigher than the one perceived by the GOP 3.2. This example proves thateven skipping the same number of B-frames, skipping them evenly producessmoother videos.

The complete algorithm to assign the importance values to the frames is foundin [9]. Also in [9] QAFS is implemented for constrained CPU and in [11] forconstrained network bandwidth.

With all priorities assigned to the frames, the selection algorithm enters a loopand discards the frames according to their priorities. The loop finishes when thenetwork bandwidth used by the adapted GOP (UNB) is less than or equal to thebandwidth available on the network. Figure 3.2 depicts this loop.

Figure 3.2.: QAFS selection loop

QAFS can be used both for online and off-line stream adaptation. An off-lineusage of QAFS algorithm can generate different quality levels video stream. Inthis case, when a user requests the video stream, the server can select the mostsuitable one according to the available network bandwidth. Also, since no realtime issues will be involved during the selection algorithm, complex strategies canbe applied analyzing larger sets of frames. On the other hand, many quality levels


will demand more storage resources for different video streams. Thus a trade-offbetween the granularity (i.e., number of quality levels) and storage resources mustbe made when adapting video streams off-line. The on-line method will adaptthe video stream according to the value of available resource in the moment ofvideo adaptation. Advantage of this approach is that the stream can be adjustedaccording to the real available resource (e.g., no approximate values), and thusgenerate a stream that “fits” more accurately in the resource availability. On theother hand, online adaptations can not afford complex selection algorithms, i.e.,should not consider a long set of frames in the selection algorithm.

3.3. Summary

Adaptation of resource utilization is a challenging problem. Mainly when bothresource demand and resource availability vary continuously and independently ofeach other. An interesting and relevant application that presents such a scenariois the video adaptation task that takes place in a digital video server. This serverhas to process the incoming video stream so that it can be transmitted over thenetwork. In this scenario the resource demands from incoming video depend onthe video contents, and the resource availability on the network status, i.e., bothvariable and independent from each other.

Some approaches to this problem have been published in the last years. Therealistic approaches for this scenario do not decode the incoming stream, but adaptit by simply skipping frames. We have shown that if we choose the frames to beskipped according to some frames’ characteristics (e.g., using QAFS algorithm),we can produce good quality output video stream. Other approaches (e.g., scalablevideo) can be chosen to adapt video input, but those are used in the practice toadapt stored video sources instead of live streams.

Figure 3.3 depicts the result of QAFS video adaptation for 6 minutes of video underconstrained network bandwidth. The green curve shows the network bandwidthrequired to send the original video, the magenta one shows a constant availablebandwidth and the blue curve shows the bandwidth used by the video stream afterskipping frames. One point still non-optimized for frame skipping algorithms: theyproduce output video streams considering the available resource as the maximumusage upper bound. It results in resource suboptimal utilization (for this simulation16 MBytes/s of bandwidth, 27 MBytes of data transmission have been wasted).In this work we propose to break this paradigm (the area in red represents thenetwork bandwidth that was not used).


0 50 100 150 200 2500

1

2

3

4

5

6x 10

5 Network BW usage

Time (s)

Net

wor

k B

andw

idth

(B

ytes

/s)

Wasted Network BWRequired Network BWAvailable Network BWUsed Network BW

Figure 3.3.: QAFS: available network bandwidth suboptimal utilization

4. Proposed Solution

In Section 3.2.2 the frame skipping algorithm used was presented, and Figure 3.3exposed that available resources are wasted. The frame skipper bounds its resourceusage to the value of the instantaneous availability resource (in some applications,this value represents a weighted mean value). In our proposal, we will relax thisimposition and allow short term resource usage overshooting. The resource usagewill be controlled using a feedback control loop.

To design this control loop, at first, there is need to model the controlled process. Inthis work, we will use the previously presented QAFS algorithm to adapt incomingvideo stream. Thus, in the following section, the QAFS video adaptation task willbe modeled as it is, considering that the QAFS stream output will be directlysent to the network. We will point out the difficulties faced to design a controlleron this approach. Afterwards, we will present an alternative scenario in whichthe controller will be implemented. In the sequel we specify the performanceparameters which must be achieved by the controlled system, and at last we presentthe controller itself.

The main target of the closed system is to adapt the incoming video stream sothat the difference between the network bandwidth used after stream adaptation(UNB) and the available network bandwidth (ANB) tends to zero.

As presented in Section 2.4, wireless networks are intrinsically not real time net-works (although some algorithms can be applied to improve their predictability).Thus, in our approach, the usage of control signals through the network will berestricted to the minimum, so that controller performance is not influenced bynetwork unpredictable behavior. In Section 4.1, the intuitive approach to modelthe video adaptation task is presented. We will explain the reasons why it fails onrepresenting a system that can be controllable. In Section 4.2 another approach isused to model the system, including a new element that permits the whole systemto be controllable.

33

4. Proposed Solution 34

4.1. One-to-one Model

To start modeling the QAFS, the video stream adaptation scenario will be mappedinto the control loop diagram. In this approach, the QAFS algorithm is the systemto be controlled, i.e., the “plant”. In the following, the plant will be modeled.

A model is an artifact (usually expressed by equations) used to reproduce thesystem behavior. For QAFS, we are interested in modeling the output/input rela-tionship. We consider the amount of network bandwidth necessary to transmit theadapted video stream as the system output and the network bandwidth availabilityto QAFS as the system input (see Fig. 4.1). To start modeling the system, it isnecessary to restrict the adaptation system by doing some assumptions. Further-more, it is necessary to map the notations presented in the video adaptation taskinto their respective pairs in the control diagram (the term “signal” will be usedfor values and measurements used in the control system diagram). To be consis-tent on the notations, we state that when the system is sampled at time t(k), theavailable signals use the notation (k), e.g. r(k), y(k) for reference and output signalsrespectively, sampled at time t(k).These assumptions and notations are presentedas follows:

1. video adaptation is started once for each GOP. Thus, whenever a new GOPis received by the stream adapter, the controller must provide a new networkavailability value. Therefore, we select the sampling period to be the same asthe GOP period (internally, the tasks run under much shorter periods). Forinstance, considering the video frame is 20 FPS, and the GOPs are 12 frameslong, the GOP period, and consequently the sampling time is 0, 6s

System sampling time = GOP period

Ts = TGOP (4.1)

2. the network bandwidth available to QAFS will be mapped as the systeminput.

u(k) = ANB∗(k) (4.2)

3. the used network bandwidth after frame skipping will be presented as systemoutput.

y(k) = UNB(k) (4.3)

4. the network bandwidth required for sending the GOP without any adaptation(required network bandwidth: RNB(k)) will be treated as disturbance signal


to the system, i.e., it will affect the manner how UNB(k) will deviate fromANB(k).

d(k) = RNB(k) (4.4)

5. the available network bandwidth ANB(k) is mapped as a set point to thesystem.

r(k) = ANB(k) (4.5)

Figure 4.1.: Map the video adaptation into the control loop diagram

With all assumptions done, notations and functionalities mapped, we can start tomodel the given system. At first, the frame skipper must be modeled. We will showthat according to the method of theoretical analysis, used to create a mathematicalmodel by means of analyzing the physical characteristics of the system, the QAFSdoes not provide any dynamic equation.

The QAFS’ output signal is equals to the sum of the required bandwidth of allframes present in the adapted GOP. These frames are selected according to theirpriorities, which were given by the priority assignment algorithm. The frame


selection loop discards the lower priority frames until the condition UNB ≤ ANB∗

is achieved (see Section 3.2.2). Indeed, the frame priorities and consequently theparameters used to define which frames are to be present in the adapted streamis undefined until the system receives the GOP. Thus, a precise QAFS modelmust take into account not only the system input, but also detailed informationconcerning disturbances. According to this model, the plant’s output (UNB) isstrongly influenced by time varying disturbances (RNB) (although they can bemeasured, they can not be predicted). We will next model the QAFS.

As we will show next, QAFS has no dynamic states. Dynamic systems are thosewhose present state depends on previous states. In the model presented in thissection, the stream adaptation that takes place for a given GOPi in the moment i,does not take into consideration any state from previous GOPs (GOPi−1,i−2,...), itsstate depends exclusively on the instant input (available network bandwidth) andinstant disturbance (input video stream). The QAFS output can be formulated asa function of its input and the disturbances applied to it:

u+(k) = ψ(u′(k), d(k)). (4.6)

Further analysis on the system behavior allows us to state that:

u+(k) ≤ u′(k). (4.7)

A simplified QAFS model used during our simulations is presented in Figure 4.2.

ANB*

1

Unity

1

Unit Delayz

1

Switch

Subtract

Relational

Operator

<

ProductDelta

3

RNB

2

ANB ’

1

QAFS

coefficient

ABW ’

Figure 4.2.: QAFS model used in simulation

The QAFS model was designed to represent the following behavior: whenever theANB(k) is greater than RNB(k), no frame is skipped, hence the QAFS outputis equals to RNB(k). On the other hand, if the incoming stream demands more


network bandwidth than the one available, the frame skipper will produce anoutput stream that requires less (or maximum the same amount of) bandwidththan the available bandwidth. In Eq. (4.8) we present the frame skipper modelused in the simulations. The QAFS’ output ANB∗

(k) is modeled as a function

ψ(u′(i)(k), d(i)(k)), described as follows:

ψ(u′(k), d(k)) =

RNB(k) if RNB(k) ≤ u′(k)

u′(k) × (1 − δ(k)) if RNB(k) > u′(k),

| 0 ≤ δ(k) ≤ |γ|, 0 < γ ≤ 1

(4.8)

Although the disturbance characteristics can be roughly modeled, it is not possi-ble to compute precisely the system’s output unless a very detailed disturbanceanalysis (size and type for all frames within each GOP) is available. Moreover,these disturbances vary drastically for every GOP according to the video contents.Table 4.1 exposes those differences for some different GOPs presented in a newsprogram.

u = ANB = 150 KInput stream size in (KBps) Output stream size in (KBps)

GOP# 29 366,660 149,800GOP# 92 406,700 117,500GOP# 160 602,540 0, no frame is transmittedGOP# 268 459,660 78,020

Table 4.1.: QAFS output variation for a news video stream with constant inputANB = 150 KBps.

Therefore, if we consider that QAFS algorithm can find the optimal adaptation forinstantaneous network availability and input video stream (i.e., QAFS algorithmcan not be further optimized), with the model presented in this section, nothingcan be controlled to improve the system’s performance. A method to overcomethis characteristic motivates the next section.

4.2. Revisited Model

If we consider the QAFS algorithm as the only component able to adjust theused network bandwidth, no further improvement can be achieved to optimize


the resource usage (here we do not want to propose improvements on the QAFSalgorithm itself, we assume the QAFS is already “tuned” to use as much networkbandwidth as possible). To insert a new degree of freedom on the system and thenimprove the network bandwidth usage, we need to include a new component.

4.2.1. Augmented model

This new component should offer possibilities to “accommodate” the difference be-tween the amount of resource available and the one used after stream adaptation.Not only “accommodate” these values, we must also be able to control this infor-mation and feed the resource availability “seen” by QAFS accordingly. To achievethose requirements we changed the original video stream adaptation scenario byinserting a buffer immediately after QAFS and before the task that will transmitthe stream over the network. Figure 4.3 shows this change and how we mappedthe whole system in the control diagram.

For this model, we will map the video stream to the control scenario as follows:

1. as discussed in Section 4.1, the system’s sampling time (Ts) must be chosensuch that the controller computes new input for the stream adaptation beforeeach adaptation task starts. Again, the system sampling time is mappedto be the GOP period1, i.e., adapt each GOP according to the networkbandwidth, as in Eq.(4.1).

2. the stream adaptation task is mapped as an actuator. It receives the signalfrom the controller and is responsible to forward it to the plant. In thisscheme the influence of video contents can be modeled as actuator distur-bance (see item 10).

3. the buffer level is the value we want to control in this scenario, i.e., the bufferis the plant. We will develop the controller (to be presented in Section 4.3)so as to keep the buffer level in a certain value. Figure 4.4 presents the FIFObuffer implemented in this work as well as all signals and data flow relatedto it.

4. the sensor is responsible to read the buffer level and convert it into therespective network bandwidth value (further details in Section 4.3.1).

BBW (k) =BL(k)

TGOP

⇔ w =y(k)

TGOP

(4.9)

1This choice does not violate the sampling theorem presented in Section 2.3. Here we considerthe ANB(k) signal varies in a frequency much lower than Ts


Figure 4.3.: Map the augmented video adaptation into the control loop diagram

5. the network bandwidth availability seen by the adaptation task is given bythe controller output value.

u′(k) = ANB′

(k) (4.10)

6. the buffer receives the contents of adapted GOP. This information is mappedinto the control loop diagram as the network bandwidth necessary to sendthis GOP. From the control system point of view, we map this signal as:

u+(k) = ANB∗

(k) (4.11)

7. for each sampling time, the amount of data correspondent to the availablenetwork bandwidth is taken from the buffer and sent over the network. Wemap the ANB(k) as:

u−(k) = ANB(k) (4.12)


8. the buffer input signal is formed by the combination of two others: thebandwidth necessary to send the adapted GOP (item 6) minus the availablenetwork bandwidth (item 7) (more details in Figure 4.4 and Section 4.3.1).

u(k) = ANB∗

(k) − ANB(k)

⇔ u(k) = u+(k) − u−(k)

9. the amount of data present in the buffer is mapped as buffer level and rep-resents the plant’s output signal.

y(k) = BL(k) (4.13)

10. the deviations caused in QAFS’ used network bandwidth originated by differ-ent video contents will be mapped as actuator disturbances, i.e., disturbanceswill influence how well QAFS can use its input bandwidth availability (u′(k)),or short how u(k) deviates from u′(k). Disturbances are represented by GOPvideo contents (and its detailed information, e.g., frame types, frame sizesetc).

d(k) = Vin(k) (4.14)

11. the buffer level set point is represented by the signal r(k)

12. and the difference between set point and the actual buffer bandwidth ismapped as the error signal e(k).

Mathematical description

Considering the augmented system, we need next to create the model for eachblock: sensor, actuator and the plant. The “property” to be controlled is networkbandwidth, for this reason we will model the system based on bandwidth signals.In the mapping presented in Section 4.2.1 item 4, the sensor model has already beenmodeled according to Eq. (4.9). Moreover, the actuator’s model, i.e., the QAFS’model, has been derived and presented in Section 4.1.according to Eq. (4.8). Wecan derive the buffer model by analyzing the difference equation that describes itsbehavior. The buffer has the same physical characteristics as a water tank. In thatcase, the mass variation in a tank is equal to the mass of water that flows in thetank minus the mass that flows out of it, i.e., the law of mass conservation. Forthe buffer, its variation rate is described by the amount of data that enters minus


Figure 4.4.: FIFO buffer. The data pushed onto the buffer by QAFS (u+) minusthe data taken from it by the transmission task (u−) is mapped into itsrespective bandwidth as buffer input for the control system (u). Thesensor converts the buffer level (BL) into the corresponding bandwidth(BBW ).

the amount of data that leaves the buffer. Let us derive this difference equation,for b being the buffer level and Φ representing data flow:

∆b(k)

∆t= Φin(k−1) − Φout(k−1) (4.15a)

b(k) − b(k−1)

Ts

= Φin(k−1) − Φout(k−1) (4.15b)

where:

Φin(k) = ANB∗

(k) = u+(k)

Φout(k) = ANB(k) = u−(k)

Applying the Z -Transform to the difference Eq. (4.15b), we have:

b(z)(z − 1) = Ts (Φin(z) − Φout(z)) (4.16)

b(z) =Ts (Φin(z) − Φout(z))

(z − 1)(4.17)

and get the transfer function:

b(z)

(Φin(z) − Φout(z))=

Ts

(z − 1)(4.18)


As we can see in Eq. (4.18), the buffer behaves like an integrator, where its inputis the difference between the data ratio that enters in the buffer and the dataratio that leaves the buffer. However, some important characteristics must behighlighted. While Φin(z) is the adaptation task output signal (controllable signal),Φout(z) can not be controllable - it can be read though (this characteristic will beused for the control design in Section 4.3.1). Furthermore, we should investigatethe buffer operation so as to make this model more accurate. One constraint thatmust be imposed to this model is not allowing the buffer model to have a negativelevel, since it can never happen in the real buffer (although buffer underflow is areal possibility, the buffer itself will never contain negative amount of data). Ifwe consider the formulated equation 4.15b, we can easily see, for instance, thatwhenever the buffer is empty, and the available bandwidth is greater then thebandwidth used by the adjusted GOP, we would have a negative value for thebuffer level.

b(k) = b(k−1) + Φin(k−1) − Φout(k−1) (4.19)

b(k) = 0 +ANB′

−ANB

b(k) < 0, if ANB > ANB′

To avoid this effect, an external artifact is used in the simulation to reset the bufferlevel to zero, whenever the integrator reaches a negative value (see Figure 4.5).

Buffer

Level

1

ZOH

Subtract

Discrete−Time

Integrator

1

z−1

Compare

to Zero

< 0

Fill in

Rate

2Fill out

Rate

1

Figure 4.5.: Buffer model with reset trigger for negative values

4.3. The Controller

The video adaptation system is modeled as presented in the previous section. Thenext step to design the controller for the buffer level is to specify the goals to


be achieved for the whole system, in other words, which performance parametershave to be matched. The video adaptation system must use the available networkbandwidth such that:

1. Overshoot is bounded

2. limt→∞

∑

(ANB(t) − UNB(t)) → 0

3. do not introduce significant delay on the adaptation process

If those specifications are fulfilled we can ensure that the resultant video playedon the client side presents better quality than the one whose adaptation systemhas no controlled buffer.

In the practice, the video adaptation system as presented in Fig. 4.3 is formedby individual tasks sharing their input/output signals through their interfaces. Inthe sequel we describe all blocks in the adaptation system, their interfaces andsampling time.

4.3.1. Detailed system description

It is important to mention that all blocks in the system are sequentially triggered,i.e., each block starts its computation when its predecessor delivers its outputsignal. The system’s computation cycle is synchronized with the “system samplingclock”. This clock is triggered with the arrival of a new GOP in the input videostream. We will present the blocks’ description on the same order as they appearin the control diagram shown in Figure 4.3(although some blocks are not explicitlyshown there), explaining all the computations that take place during one sampleperiod.

Before starting the adaptation, the video server and client execute initial transac-tions and agree on the maximum frame rate to be used in the adaptation task (thisvalue is imposed by the client according to its constraints). Assume one GOP hasjust been received, these are the computations present on the video adaptationsystem:

Video stream input task

This task reads in the video stream and extracts from incoming stream, for eachGOP, the information necessary for the next tasks, e.g., frames types, frame sizes.When this task is complete, it triggers the system clock.


ANB reading task

This task runs the Packet Transmission Rate (PTR) algorithm [11], [7] to computethe available network bandwidth. PTR belongs to the category of Probe RateModel, thus it continuously transmits and receives packets over the network. Thesepacket transactions occur in periods shorter than the one used to synchronizethe whole system, thus for each system clock a new computed value for ANB isavailable and must be read.

Set point

A new set point value must be acquired for each sampling time. More about setpoint computation in Section 4.3.3.

Controller

Here we assume the summator presented in Fig. 4.3 is part of the controller. Thecontroller reads the set point and feedback values and computes the controlleralgorithm to generate the bandwidth value to be used by QAFS for every GOP.The controller’s output is available to the actuator on the same clock sample.

Actuator

QAFS algorithm runs internal loops for priority assignment and frames discarding.It finishes its execution in time to deliver the adapted video stream to the bufferin the same clock period. The contents of adapted video are then stored in thebuffer.

Sensor

After QAFS pushes the adapted GOP into the buffer, the sensor reads the amountof data present in the buffer. It is not a usual technique to measure this value atthis point in time, before the whole cycle is completed, i.e., before data is extractedfrom buffer. The reason why we adopted this approach will be explained in 4.3.3.The sensor’s function is to convert the buffer level (BL) value into its correspondingbandwidth (BBW ) (see Eq.(4.9)).


Plant

The FIFO buffer stores the adapted video contents before the transmission taskremoves it to send it over the network. From the buffer’s level point of view, theplant represents an integration of the difference between the size of the adaptedvideo to be stored in the buffer and the amount of data to be extracted from it bythe transmission task. For discrete time system, the integration:

∑

Ts (Φin(k) − Φout(k)) (4.20)

is represented by Eq. (4.17).

Transmission task

This task is executed after the sensor reads the buffer contents’ level value. Atevery sampling time, the amount of data corresponding to the available networkbandwidth is read from buffer and sent over the network:

amount of data to send = ANB × TS (4.21)

After the transmission task takes the data from buffer, the system cycle is complete.A new cycle starts when the video stream input task extracts the next GOP.

4.3.2. Assumptions

As we mentioned in Section 3.1, the scenario in which the controller is insertedcontains many unpredictable variables, e.g., incoming video stream, network avail-ability fluctuations, CPU availability for the video adaptation task, client videobuffer size etc. In the following, we will list the assumptions that we have doneto develop the controller presented in this work. They are consistent assumptionsand do not impose unreal constraints to the video adaptation system:

Unconstrained CPU : We assume the hardware used in this work does not imposeany constraint concerning the computational power necessary to run thevideo adaptation system. The only constraint applied to the system is thenetwork bandwidth availability.

ANB is a smooth signal : In Section 2.4 we explained that the network band-width availability varies continuously and can present high frequency varia-tions. In this work, this high frequency variations will not be considered since


they do not represent “structural” availability variation. The ANB readingtask will provide an availability measurement already low-pass filtered andthose high frequency variations will not be present.

ANB value is not accurate : The network bandwidth availability is a discreterepresentation of a continuous value, and it contains intrinsically small de-viations from the real value. As we mentioned in Section 4.3.1, the ANBreading task runs in a period shorter than the system clock. Thus, ANBmeasurements done very close to the instant system sampling time tend to bemore accurate than the ones done immediately after the previous samplinginstance.

Bounded overshooting in the transmission rate is allowed : We can afford tohave short term and bounded overshooting on the network transmission rate.As we mentioned previously, the ANB value measured is not accurate, thusnot using network bandwidth beyond ANB may lead to suboptimal networkusage. In case this overshooting represents significant load on the network,it will be reflected on the next ANB measurement.

Player on the client can handle 2 GOPs : We assume the client buffer does notoverflow unless more than to 2 GOPs are sent in one GOP period. Thisassumption allows us to transmit more than one whole GOP in a certainGOP period, in case the available bandwidth and the server buffer contentspermit it.

4.3.3. Controller design

Structure

The controller used in this work has a structure slightly different from the onepresented so far. For the structure presented in Figure 4.3, when the systemoutput reaches the set point, the controller’s output is equal to zero, it wouldmean, the stream adaptation task should use network bandwidth equals to zero toproduce the adapted stream. This would result in an empty GOP.

u′(k) = 0

⇒ u+(k) = 0(4.22)

In a correct scheme, for zero error, the stream adaptation input must receive the


Figure 4.6.: Signal tracking control for buffer bandwidth BBW

current available network bandwidth to adapt the current GOP:

u′(k) = ANB(k)

⇒ u+(k) = ψ(u′(k), d(k))(4.23)

Thus, in this work we use the tracking control scheme, as presented in Figure 4.6.

We will control the buffer bandwidth of the system using the scheme presented inFigure 4.6. Plants whose transfer functions present one pole at the origin (as isthe case of an integrator), should present no steady-state error: ǫss = 0. But thisis not what happens in this system. Indeed, equation 4.18 may lead us to concludethat the plant’s transfer function has the pole at the origin. However, a closer lookin the system (and Eq. (4.18)) will reveal that the plant’s input signal is not theone that comes from the actuator, instead it is the combination of the actuator’soutput minus u−. Thus, so as to achieve a system with no steady state error, acompensator with a pole at the origin must be used in this controller’s design.

Set point

The controller’s goal is to guarantee that the stream adaptation task will producea video stream such that it will use all available network bandwidth. A naıveapproach is to set the buffer set point to zero and ensure the buffer is emptyafter the whole system cycle is completed: GOP detection, control computation,stream adaptation, push buffer, pop buffer. Another approach is to ensure that,after pushing the GOP into the buffer (immediately before the transmission taskrequires the data to transmit), the amount of data stored there corresponds todata that can be sent using the available network bandwidth:

BL(k) = ANB(k) × Ts(k) (4.24)


Next we explain why we have chosen this unusual approach. The more intuitiveapproach would be to measure the buffer level after the transmission task popsdata from the buffer: the system’s tasks cycle finishes after data transmission.The problem that arises, is that, either if the ANB is exactly equals to BBW

(bandwidth corresponding to the buffer level) or for instance, ten times greaterthan that, the buffer will be empty afterwards, i.e., the feedback value will bezero. In the sequel we show one example that explains this case.

Let us consider that the system has just finished all computations for a given timetk, and the results are presented in the Table 4.3.3:

GOP(k), t = t(k)

r(k) 100 KBpsǫ(k) 10 KBps

u′

(k) 8 KBps

ANB(k) 100 KBpsu∗(k) 108 KBps

u+(k) 90 KBpsu−(k) = ANB(k) 100 KBpsBBW (k) 1 KBps

Table 4.2.: System snapshot after adaptation of GOP(i), immediately before push-ing the GOP into the buffer.

In this example, the buffer has data level corresponding to 1 KBps, it will receive90 KBps from the adaptation task and the transmission task will take 100 KBpsfrom it. For the first approach, the buffer will be empty after the transmission tasktakes the data, and the measured value will be zero - the measured value would bethe same if ANB was 3.000 KBps or exactly equal to 91 KBps. For this example,the ideal measured value should inform that, after this time clock, the buffer “has”9 KBps non used bandwidth.

To overcome this problem we store the signal containing the buffer size before thetransmission task takes data from it - we will call this signal as B−

L(i). In this case,for the previous example, the buffer bandwidth measured value is:

B−

BW (k) = BBW (k−1) + (u+(k) − u(k)) (4.25)

B−

BW (k) = 1 + (90 − 100)

B−

BW (k) = −9

In this approach, we must choose the buffer set point to be equal to the available


network bandwidth (ANB(i)). Consequently, for the steady state:

B−

BWss = ANB(k) (4.26)

To summarize, the controller will compare the value that was present in the bufferbefore the transmission task pops the data, with the available network bandwidthof the previous sample and apply to it the control algorithm:

ǫk = B−

BW (k−1) −ANB(k−1) (4.27)

Control algorithm

In this work we use the widely studied PID algorithm as buffer’s bandwidth con-troller. This controller can be seen as the sum of three different terms that willoperate on the error signal according to their respective coefficients: multiplicative(Kp), integral (Ki) and differential (Kd). [4] presents further discussion concerningPID controllers theory. As described in previous sections, the controller’s inputand output signal are represented in this work as ǫ and u

′

respectively. The con-tinuous controller equation and its respective transfer function are presented inEq. (4.28a) and Eq. (4.28b).

u′

(t) = Kp ǫ(t) +Ki

∫ t

0

ǫ(τ) dτ +Kd

dǫ(t)

dt(4.28a)

U′

(s)

E(s)=

Kd (s2 + Kp

Kds+ Ki

Kd)

s(4.28b)

The difference equation used to implement the algorithm is presented in Eq. (4.30)

u′

(k) = u′

(k−1) + (Kd

TS

+Kp +KiTs) u′

(k) (4.29)

−(2Kd

TS

+Kp) u′

(k−1) +Kd

TS

u′

(k−2)

The next step is to find the three coefficients used in the PID controller such thatthe controlled buffer can achieve the given set point with small overshooting, fastrise time and settling time and low oscillation, satisfying the goals presented inthe beginning of this section (Pag. 43). We need to find a trade-off among thoseparameters and then select the coefficients Kp, Ki and Kd accordingly. Table 4.3shows the effects of changing those parameters. The effects explained in the tableassume the increase of the correspondent parameter:

We use the Ziegler-Nichols method to set those parameters. In this method we startsetting all parameters to zero, and then start increasing the proportional gain until


Effects for increasing parameters

Parameter Rise Time Overshoot Settling Time SS Error

Kp Decrease Increase Not significant DecreaseKi Decrease Increase Increase EliminateKd Not significant Decrease Decrease None

Table 4.3.: Effects for increasing PID parameter

the system oscillates. This gain is called critical gain (Ku). The oscillation period’snotation is Pu. After this practical experiment, we can set the PID coefficientsaccording to the Table 4.4:

Ziegler-Nichols coefficients for PID controllers

Kp Ki Kd

0.6Ku 2Kp/Pu 0.125KpPu

Table 4.4.: Ziegler-Nichols method

According to the information given in the two previous tables, we initialize thePID parameters according to the Ziegler-Nichols method and tune them using theTable 4.3 to improve the performance parameters. In the following section wepresent the practical procedures to find out those values.

4.3.4. Simulation and implementation issues

System simulation

The complete system (on the video server side) was simulated using Matlab/SimulinkR2007b installed under Linux on an Intel Pentium Dual Core 3.20 GHz, with 2 MBcache and 1 GB RAM machine. All detailed Simulink blocks used for this simula-tion are shown in the Appendix A.3. Next we present our simulated system.

Figure 4.7 shows the outer blocks used in this simulation. This figure shows thatthe simulation maps one to one the system presented in Figure 4.6. As mentionedin Section 4.2, the QAFS’ output stream is a function that depends not only onits input from the controller, but also on the incoming video stream contents.To simulate the frame skipping behavior, we use the previously mentioned model(Fig. 4.2) and apply to it the signal γ as a randomly generated value. Doing so,we assume that, whenever the QAFS skips frames, its output stream will demanda random portion of the available network bandwidth. Practical experiments have


shown that the frame skipper produces video streams that use, in most of thecases, 100% to 90% of the available network bandwidth, thus we choose γ to varybetween 0 and 0.1. The set up of the controller’s parameter is depicted in the

ZOH3

ZOH2

ZOH1

Uniform Random

Uniform

Random

Number

Step

Repeating

Sequence QAFS

Model

ANB ’

RNB

Gamma

ANB*

PID−Controller

with antiwindup

E act

_IN

act

_O

UT

AN

B

RN

B

Se

t−p

oin

t

U

Delay

z

1

Constant1

.1

Constant

2E5Buffer

Bandwidth

Sensor

In1

In2Out1

Buffer

Fill out Rate

Fill in Rate

Buffer Level

RNB

ANB

ANB

Feedback

Gamma

error

Fill in Rate

Figure 4.7.: Simulink model: system overview

sequel. In the next examples, we enforce the input video stream to require muchgreater network bandwidth than the one available on the network:

RNB(i) ≫ ANB(i) (4.30)

Figure 4.8 shows the experiment’s results used to select the controller gains, ac-cording to the Ziegler-Nichols method. Those are presented in Table 4.5

Controller gains

Kp Ki Kd

0.4 1.33 0.05

Table 4.5.: Coefficients for PID controller

Figures 4.9(a) and 4.9(c) show the system response for a simulated available band-width in the form of a unit step and considering that the frame skipper’s outputstream requires fixed 90% of the bandwidth given to it. The controller’s param-eters are set according to the Ziegler-Nichols method. For those coefficients, the


(a) Unstable output forKp = 1.2

(b) Stable output forKp = 1.0

Figure 4.8.: Change Kp until the limit when the system oscillates. Step signalapplied to the input has amplitude of 1 × 105

closed loop system presents 5% overshoot and settling time 4.2s for 95% of steadystate amplitude. With small changes in the controller coefficients, we can achievesettling time of 1.8s for 95%, and no overshoot (Figures 4.9(b) and 4.9(d)).

In Figure 4.10(a) and Figure 4.10(b) we show the result of the simulation for a morerealistic scenario, where the frame skipper produces the video stream according toEq. (4.8) where 0 ≤ γ ≤ 0.1. These figures show the comparison between the twoset of PID coefficients. Figure 4.10(b) present smaller overshoot, as expected forthe tuned parameters. Therefore, the tuned coefficients are used on this work forthe next experiments.

As we mentioned in Section 4.3.2 Assumption“ANB is a smooth signal”, we assumethat the available network bandwidth value will not present abrupt changes, thuswe present the system response for a smoothly varying ANB in the next figures. InFigure 4.11(a) and Figure 4.11(b), we still impose great RNB values, for smoothANB variations.

In Figure 4.14(a) and Figure 4.14(b) we reduce the RNB to show the windupeffect. In these figures, the RNB is presented in green. The windup effect takesplace when, even though the controller imposes a high input value to the actuator,the actuator can not pass such a high signal to the plant, i.e., it behaves as asaturated actuator whose maximum value is the RNB. Mapping this effect tothe adaptation stream scenario, it means that the controller gives a high networkbandwidth value to be used by the stream adapter, however, since the incomingstream does not demand so much bandwidth, it does not send so much data tothe buffer. The problem occurs because the controller keeps integrating the high


(a) Step response (b) Step response with tunedcoefficients

(c) Detail: settling time of4.2s for 95%

(d) Detail: no overshoot andsettling time of 1.8s for 95%

Figure 4.9.: The system with gains selected according to Ziegler-Nichols are shownin 4.9(a) and4.9(c). The same experiment, for tuned coefficients(Kp = 0.4, Ki = 0.667 and Kd = 0.05) in 4.9(b) and 4.9(d)

value it sends to the actuator (not the one actually used by the plant). Afterwards,whenever RNB rises again, the high value accumulated on the integrator appliedto the actuator can then be passed to the plant, and a huge overshoot takes place.Therefore, a mechanism (called anti-windup) is applied to the controller to avoidthis effect.

The effect of anti-windup is shown in Figure 4.15(a), and in Figure 4.15(b). In Fig-ure 4.13 and Code 1 we present the anti-windup algorithm as well as its connectionwith the PID controller.

The windup effect takes place whenever there is a difference between the actuator’sinput and output. Indeed, this difference tends to be very high for the caseswhen RNB < ANB, and thus need to be strongly compensated. Nevertheless,


(a) Coefficients according toZiegler-Nichols

(b) Optimized coefficients

Figure 4.10.: System response for constant ANB and 0 ≤ γ ≤ 0.1 (larger simula-tion time: 600s)

(a) Response for constant γ =0.1

(b) Response for 0 ≤ γ ≤ 0.1

Figure 4.11.: More realistic ANB simulation: smooth signal

windup effect also occurs for the mostly common cases, where RNB > ANB.In this case, strong antiwindup would affect the steady state error. To overcomethose two “operation modes” we use two different antiwindup coefficients: one forRNB < ANB and other for RNB > ANB.

In Figure 4.14 and Figure 4.15 we compare the controller signals for a system withand without anti-windup. As we can see, the anti-windup algorithm reduces theactuator input signal when ANB > RNB. This is achieved because the integratedvalue is reduced, thus when ANB < RNB again, the plant will not receive a largesignal.


State

2

Output

1

Zero−Order

Hold3

Zero−Order

Hold2

Zero−Order

Hold1

Gain

c3

c3

Gain

c2

c2

Gain

c1

c1

Delay−

Computation

Time4

z

1

Delay−

Computation

Time3

z

1

Delay−

Computation

Time

z

1

Antiwindup

2

Error

1E(k)

E(k−1)

E(k−2)

Figure 4.12.: PID model used in the simulation

U

1

PID

controller

Error

Antiwindup

Output

State

Antiwindup

State

ANB

RNB

act_IN

act_OUT

Out1

RNB

5

ANB

4

act_OUT

3

act_IN

2

E

1

(a) Controller with antiwindup

Out1

1

ZOH2

ZOH1

Embedded

MATLAB Function

statevar

act_in

act_out

anb

rnb

awu

y antiwindup

Delay−

Computation

Time

z

1

Anti−windup

coefficient

awu

act_OUT

5

act_IN

4

RNB

3

ANB

2

State

1

(b) Antiwindup

Figure 4.13.: Antiwindup connected to the PID controller

Implementation issues

The PID controller presented in the previous section, is incorporated in the frameadaptation project designed in [11]. During the controller’s implementation, someimprovements have been done also in the frame adaptation code, mainly to syn-chronize threads that run concurrently during the whole execution of the system.However, details concerning the preexisting code are not in the scope of this thesisand will not be further analyzed. The code incorporated in the frame skipperproject is separated in three different categories: one to acquire the data to beused by the controller, e.g., available network bandwidth, adapted video stream;one to compute the controller algorithm; and the third one to collect the relevant


Code 1 Anti-windup algorithm implemented in Simulink

function state_out = antiwindup(statevar, act_in, act_out, anb, rnb,

awu)

% Antiwindup:

if ( act_out < act_in ) % normal operation

if (rnb < anb)

state_out = (act_in - act_out) * awu;

else

state_out = (act_in - act_out) * awu / 15;

end

else % act_in is negative, act_out is zero

state_out = statevar; % cancels out the state feedback

end


(a) Presence of windup for0 ≤ γ ≤ 0.1

(b) Detailed view of windup

Figure 4.14.: Controller signals for a system without anti-windup mechanism. Thesignals presented here are: set point in red, controlled buffer band-width in blue, actuator input in magenta, actuator output in cyanand RNB in green

.(a) Anti-windup for 0 ≤ γ ≤0.1

(b) Detailed view of anti-windup mechanism

Figure 4.15.: Anti-windup acting in the controller. The same color pattern asin 4.14


Code 2 PID with anti-windup

void Pid::execute(float ts, float actuatorOutputPrev, float ANB,

float RNB) {

if (Ts != ts) {

computeocompactcoeff(ts, &c1, &c2, &c3);

}

controlOutputPrev = controlOutput[0];

sum1.set_blockInput(setPoint[0], 0);

sum1.set_blockInput(-feedback[0], 1);

sum1.execute();

error[0] = sum1.get_blockOutput(0);

// antiwindup

if (RNB < ANB) {

antiwindup = ( (controlOutputPrev + ANBPrev) - actuatorOutputPrev )

* Kaw;

} else {

antiwindup = ( (controlOutputPrev + ANBPrev) - actuatorOutputPrev )

* (Kaw /15.0);

}

// errorPrevPrev = err$(k-2)$

// errorPrev = err(k-1)

// error[0] = err(k)

// controlOutputPrev = u(k-1)

// = PID =

controlOutput[0] = controlOutputPrev + error[0]*c1 + errorPrev*c2

+ errorPrevPrev*c3 - antiwindup;

errorPrevPrev = errorPrev;

errorPrev = error[0];

ANBPrev = ANB;

}

void Pid::computeocompactcoeff(float ts, float *c1, float *c2,

float *c3) {

Ts = ts;

*c1 = Kd/Ts + Kp + Ki*Ts;

*c2 = -(2*(Kd/Ts)+ Kp);

*c3 = Kd/Ts;

}


data to generate analytical graphs. They were incorporated in the code so as to ac-complish the execution sequence explained in Section 4.3.1. In Code 2 we presentthe code that executes the controller algorithm.

4.4. Summary

The video adaptation task alone can not be controlled with satisfactory resultsbecause it is strongly influenced by external disturbances and it does not haveany dynamic state (the instant state does not depend on the previous one). Toovercome those characteristcs we introduced one buffer into the system. It islocated immediately after the frame skipping task and before the transmissiontask. The frame skipper and buffer tasks are mapped into the control diagram asactuator and plant respectively.

The buffer’s and QAFS’ mathematical model as well as their practical constraintswere presented in details. We also presented how the signals are passed from onetask to another. We have shown that the buffer bandwidth BBW (amount of datastored in the buffer that will be sent within one sampling period) is the controlledvalue. This signal is measured before the transmission task takes data from buffer(and not after the cycle completion), so as to allow realistic feedback values to besent to the controller (see 4.3.3). The controller’s set point is then the networkavailable bandwidth, and the controller will run so as to keep the buffer with theamount of data that corresponds to the available network bandwidth.

In this work we use a PID controller with anti-windup to controll the buffer band-width. This controller can be seen as the summ of three controll techniques:proportional - that acts in the instantaneous error value, integral - that influencesthe accumulated error, and the derivative one - that acts on the error variation val-ues. To tune this controller, we initialize the gains according to the Ziegler-Nicholsmethod. Afterwards we fine-tune them observing the simulation results.

The controller designed here is implemented in C++ and incorporated in the frameskipper code designed in a former project. In Figure 4.16 we show the differencein the network bandwidht usage and on the number of transmitted frames for a 6minutes incoming video, under constant available bandwidht, for a system withoutand with controller. Details concerning this comparison and the results achievedby this work are presented in the next chapter.


−2

0

2

4

6

8

10Difference Between Transmitted Frames for Systems With and Without Controller

Fra

mes

time (s)

0 50 100 150 200 250 300 350

−5

0

5

10

15x 10

4 Difference Between Transmitted Data for Systems With and Without Controller

Bps

time (s)

Figure 4.16.: Network bandwidth usage improvement using the designed PID con-troller. Positive values indicate increase on the bandwidth usage andon the number of transmitted frames

5. Results

5.1. Experiments

In this chapter we present the results collected from the video stream adaptation forthe systems with and without buffer controller. Throughout this chapter it will bepossible to evaluate the improvements generated by the buffer controller. The setof experiments have been done by applying four different incoming video streams:one variety program, one volleyball match, one news program and one cartoon. Foreach of these, we simulated the ANB as a smooth curve, with five steps, whereeach step transition is performed by a ramp. Although these ANB curves haveroughly the same shape, they were adjusted so that they impose different behaviorfor the frame skipper and the buffer controller for each video stream.

All four incoming video streams are 6 minutes long, composed by 7200 frames. Be-fore starting to analyze the results, we will discuss some important characteristicsof the selected incoming video streams.

5.2. Incoming video streams

Except for the news video stream, all others incoming video streams are formedby frames with 720 × 576 pixels. The news program is composed by frames with480 × 576 pixels. This difference will affect the absolute size of the video stream,but in our experiments we are interested in analyzing how the video streams andthe ANB variations impact on the adapted video stream. Thus the absolute framesize does not play an important role.

The news stream presents a stable demand of network bandwidth. During most ofthe time, the video stream is composed by a static background, the news anchor,the news ticker running on the bottom, and some related graphs presented duringan interview. The main characteristic of this kind of stream is the small sized P -and B-frames. Indeed, they don’t have to represent big movements on the screen,since for the most of the time, only the face of the news anchor is moving. The

61

5. Results 62

volleyball stream, on the other hand presents a much more dynamic content, whereplayers are moving all the time. It generates big P- and B-frames to represent thosemovements. Although, it can be observed that at around the time of 80 secondsthe bandwidth demand decreases for around 20 seconds. In this moment, thevideo shows a pananoramic view of the volleyball court. This kind of content canbe very efficiently compressed, considering the volleyball court is composed by twobig rectangles with two colors. Therefore, low network bandwidth is needed tosend this part of the stream. The variety program stream shows a comparison testamong two cars. It mixes very dynamic moments when the cars are moving faston the test track with static ones where the cars are carefully shown while they areparked in a parking lot. The last stream analyzed is a cartoon. This kind of streamhas a very specific characteristic: it has very small P - and B-Frames. Cartoons,mostly, keep a very similar background while the characters move. Meaning thatthe great differences seen on the required network bandwidth are mostly causedby intra frame compression.

5.3. Results analysis

The results presented on this chapter show that for all kinds of video contents, theusage of buffer controller increased the number of frames transmitted. Further-more, for all experiments whose the ANB are at least half on the RNB (this isa rough approximation, the cause for this effect is explained later in this section),the buffer controller did not impose delays greater than one sampling period. Forthe lowest ANB, the maximum delay imposed was two sampling periods.

Analyzing the next graphs, at first look, an undesired effect takes place on thesesimulations, showing some sudden decrease on the buffer level. They occur becausethe available network bandwidth is greater than the required one. In those cases,there is nothing the controller can do to increase the network usage utilization(see i.e., at around second 100 in Fig. 5.6) (we discuss alternatives to use this“wasted” network bandwidth on Chapter 6). We observed that the best resultsachieved by the buffer controller are presented on a scenario where ANB is greatenough to send the I-frame, and the P - and B-frames are not very large. Insuch a scenario, the small increments on the bandwidth given by the controller tothe frame skipper can produce significant increase on the number of transmittedframes. That is the reason why the best improvements are present on the cartoonvideo stream. Figures 5.17, 5.18, 5.19 depict this effect. There we can see thatsmall increase on the transmitted data can achieve large increase on the framespresent on the adapted video stream. Indeed, this is the significant improvement

5. Results 63

observed by the client: increase on the number of received frames. On the otherhand, we observe large oscillations on the buffer size for scenarios where the ANBis barely enough to transmit the I-frame (see Figure 5.20). That can be explainedby the fact that for some time, there is not enough bandwidth to send even oneframe. This error is fed back and it will be used to generate a small increment onthe bandwidth received by the frame skipper, which might be enough to transmitthe large I-frame and some smaller ones. This creates the oscillation behavior.Under these oscillations, can be perceived the largest delays introduced by thebuffer controller.

From the point of view of the controller itself, we can observe that the value ofthe buffer content is controlled with no stationary error, whenever RNB > ANB(best observed in Figure 5.15). Also, it responds rapidly to changes on the contentof the incoming video stream. Furthermore, the antiwindup mechanism efficientlyavoids the controller to generate large signals for both cases when RNB >ANBand RNB<ANB.

To evaluate how good the resultant adapted video stream is, we also compare thevalue of the peak signal to noise ratio (PSNR) of both adapted video streams(with and without buffer controller) for each experiment. More details about themeaning and how to compute the PSNR are presented in Appendix A.1

5.4. Graphs

In this section we present the graphs collected from the previously mentioned ex-periments. For each experiment, five graphs and one table are presented. Thegraphs depict respectively: ANB and RNB; the increase on the transmittedframes caused by the buffer controller; the increase on the used network band-width used to transmit the stream generated by the system with buffer controller;the buffer set point and its state; and the signals read and generated by the con-troller. It is relevant for the graphs interpretation, to mention that they are notpresented on the same scale. Different scales have been chosen to show more rel-evant information according to each video stream. The table shows summarizedinformation concerning the comparison of resultant streams. In the sequel wepresent the graphs and tables.

5. Results 64

0

2

4

6

x 105

Bps

Available and Required Network BW

Available BWRequired BW

−2

0

2

4

Difference Between Transmitted Frames for Systems With and Without Controller

Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps

Buffer LevelBuffer Set Point

0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals

Available BWAdaptation InputAdapted GOP BWController Error

# of discarded frames on the adapted video stream. The complete stream is 7200 frames long

without buffer controller 2,219 with buffer controller 2,081

Reduction on # of frames skipped, compared toframe skipping with no buffer control 6.22 %

Increase on # of frames present on adaptedstream, compared to the complete video stream 1.92 %

PSNR for the adapted video stream

without buffer control 76.29 dB with buffer control 77.80 dB

PSNR gain 1.51 dB

Figure 5.1.: Comparison between stream adaptation with and without buffer con-troller. The used video content is a part of a variety program

5. Results 65

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.11 dB


5. Results 66

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 1.15 dB


5. Results 67

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 1.34 dB


5. Results 68

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 0.88 dB

Figure 5.5.: Comparison between stream adaptation with and without buffer con-troller. The used video content is a part of a variety program ramp3

5. Results 69

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6

x 105 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6

7x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.06 dB

Figure 5.6.: Comparison between stream adaptation with and without buffer con-troller. The used video content is a part of a volleyball match

5. Results 70

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6


Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6

7x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.88 dB


5. Results 71

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6


Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6

7x 10

5

Time in s

Bps

Controller Signals








PSNR gain 1.00 dB


5. Results 72

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6


Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6

7x 10

5

Time in s

Bps

Controller Signals








PSNR gain 1.53 dB


5. Results 73

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6


Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6

7x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.48 dB

Figure 5.10.: Comparison between stream adaptation with and without buffercontroller. The used video content is a part of a volleyball match

5. Results 74

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 1.01 dB

Figure 5.11.: Comparison between stream adaptation with and without buffercontroller. The used video content is a part of a news program

5. Results 75

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.10 dB


5. Results 76

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.19 dB


5. Results 77

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 1.43 dB


5. Results 78

0

2

4

6

x 105

Bps



−2

0

2

4


Fra

mes

−1

0

1

2x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.50 dB


5. Results 79

0

2

4

6

x 105

Bps



0

5

10


Fra

mes

−1

0

1

2

3

4x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 1.81 dB

Figure 5.16.: Comparison between stream adaptation with and without buffercontroller. The used video content is a part of a cartoon

5. Results 80

0

2

4

6

x 105

Bps



0

5

10


Fra

mes

−1

0

1

2

3

4x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.69 dB


5. Results 81

0

2

4

6

x 105

Bps



0

5

10


Fra

mes

−1

0

1

2

3

4x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.04 dB


5. Results 82

0

2

4

6

x 105

Bps



0

5

10


Fra

mes

−1

0

1

2

3

4x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.74 dB


5. Results 83

0

2

4

6

x 105

Bps



0

5

10


Fra

mes

−1

0

1

2

3

4x 10


Bps

0

2

4

6x 10

5 Buffer Bandwidth

Bps


0 50 100 150 200 250 300 350

−1

0

1

2

3

4

5

6x 10

5

Time in s

Bps

Controller Signals








PSNR gain 2.58 dB


6. Conclusion and Future Work

In this thesis we have proposed the use of control theory approaches to improve theutilization of constrained resources in a real time embedded system. The scenariochosen was the adaptation of MPEG-2 video streams under constrained networkbandwidth. In this system both resource demands and resource availability varycontinuously.

We have selected the QAFS as the frame skipping algorithm to adapt the videostream. Moreover, we have shown that this algorithm sub-optimally uses the net-work bandwidth because it strictly bounds the resource usage to the instantaneousmeasured value of the available resource. In our approach we have proposed torelax this imposition.

Later, we have proved that the stream adaptation using QAFS, “as it is” can notbe improved by means of a control loop since the frame skipper itself does notpresent any dynamic state, i.e., its state does not depend on any previous input orprevious state. Therefore no controller can efficiently improve the utilization of theresource usage . To overcome this lack of dynamic behavior, we have proposed theuse of a component that permits temporary storage of the resultant adapted videostream. Hence, a buffer has been placed between the adaptation and transmissiontasks. For this scenario, the goal is to maintain the buffer with as much data asit can be transmitted by the transmission task. The amount of data presented onthe buffer is then measured and converted into the corresponding bandwidth, asshown in Eq. (4.9). The system’s set point is the available network bandwidth.Furthermore, the tracking control scheme has been used so that the controlleddifference between the set point and the buffer bandwidth is added to the presentANB (Figure 4.6).

The PID controller used in this thesis has been implemented with an antiwindupmechanism. Indeed, small windup is present in every sampling period, since theactuator’s output (QAFS’ output) is always lesser than (or maximum equal to) itsinput. Therefore, two different coefficients for the antiwindup have been chosen:one for the “normal” frame skipping operation when the RNB > ANB and otherfor the case when the windup effect must be strongly compensated: RNB < ANB.

84

6. Conclusion and Future Work 85

Results achieved by the proposed method have shown the reduction of 5% to 10%on the number of frames skipped in the adapted stream, for the most of experi-ments. Moreover, the PSNR gains for these experiments have reached values closeto 3dB. Furthermore, no long delays are present in the streams generated by thesystem with the buffer controller. Indeed, only one experiment (Figure 5.20) haspresented a two sample periods delay for 2 GOPs, while all others have insertedeither one sample period or no delay at all. Our approach presents the best perfor-mance when the input video stream is composed by small P - and/or B-frames. Forthese cases, small increases on the network bandwidth given to the frame skipperby the controller produces a large increase on the number of frames present on theadapted video stream. This effect is explicit, for instance, on Figure 5.19 for thetime between 230 to 280 seconds.

It has been proved that the use of the available resources can be optimized using acontroller to adjust the resource availability perceived by the system, in scenarioswhere bounded overshoot on the resource utilization is allowed. However, somefree room still open for improvements. For instance, whenever RNB is lesser thanANB, network bandwidth is not being totally used. In these cases, if incomingvideo sources are available with different resolutions, one can increase the videoresolution sent to the client. Also, if the incoming video is not a live stream and theclient buffer supports it, more frames can be sent to the client. In this thesis, weconcentrated our efforts in controlling the buffer level under constrained networkbandwidth. Future works may consider controlling a Quality of Service (QoS)parameter instead. Concepts of QoS include not only the number of frames presenton a video stream, but also, e.g., the frame sizes, how often changes on the frameresolution and frame rate occurs (e.g., one might get a better QoS if a lower andconstant frame rate is kept, rather than switching between frame rates frequently)etc . [1] have shown methods to compute QoS for video streams considering theseconditions. In addition, more constraints may be taken into consideration to selectthe system’s set point, e.g., energy consumption on a portable device. For instance,it would be more meaningful to receive the whole news program with low resolutionthan one third of it in high resolution.

A. Appendix

A.1. PSNR

Evaluation made by human beings about the quality of a video stream is usuallysubjective. If some person is asked about the quality of video stream, answers like“good”, “not so smooth”, “not so sharp” are expected. However, it is imperativeto quantify this perceived quality so as to allow computer systems to evaluatethe quality of video streams. This quantified evaluation can be used for instance,to analyze the degradation in the video stream adaptation process. With suchvalues, it is possible to benchmark adaptation methods, to adjust video adaptationalgorithms etc. To generate a metric that expresses this quantification, manymethods are available, e.g., moving picture quality metric (MPQM), digital videoquality (DVQ), Double Stimulus Continuous Quality Scale, peak-signal-to-noise-ratio (PSNR) [15]. In this thesis we selected the PSNR as the method to computethe degradation index for the adapted video stream. We compute the PSNR forall displayed frames present in the video stream, comparing the original incomingframes with the ones that will represent the resultant adapted video stream. Inthis analysis, we assume that the video player will maintain the original framerate and therefore a skipped frame will be replaced by the last displayed frame.Figure A.1 depicts how this comparison occurs: frame 2 is skipped, thus the playerwill replaced it by frame 1; the same happens with frames 6, 7, 8, they will all bereplaced by the frame 5. The PSNR value is then computed frame by framecomparing the frames from the first row with the ones from the third row, i.e.,frame 1 will be compared with frame 1

′

, 2 with 2′

and so on.

Because the analyzed signals can be presented in a large dynamic range, PSNRis computed in the logarithmic scale. To compute the PSNR, at first the mean-square-error is calculated for the whole picture: the frame is scanned pixel bypixel. The square values of the difference between each pixel in the original andthe adapted frame is accumulated and divided by the total number of pixels in theframe Eq. (A-1). The PSNR for each frame is computed as presented in Eq. (A-2).The values presented on the tables in Chapter 5 are the mean of all PSNR values foreach video stream. A straightforward analysis of Eq. (A-2) leads to the conclusion

86

A. Appendix 87

that the higher the PSNR value, the better the adapted stream. Per definition,the maximum possible PSNR value is 100.

MSE =1

MN

∑

M,N

(I1(m,n) − I2(m,n))2 (A-1)

PSNR = 10 log10

(MAX2

MSE

)

(A-2)

where MAX is the greatest value for one pixel, e.g., for 8 bits representationMAX = 255.

Figure A.1.: PSNR computation: comparison frame by frame. In the first row arethe frames present on the incoming video stream that arrive on thestream adaptation system. The second row shows the frames thatwere discarded by the stream adaptation, and the last one presentshow the adapted stream is displayed by the video player on the client.

A. Appendix 88

A.2. MPEG-2 Profiles

ProfilesSimple Main 4:2:2 SNR Spatial High

High

4:2:0 4:2:0 or1920x1152 4:2:290 Mb/s 1920 x 1152

100 Mb/s4:2:0 4:2:0 4:2:0 or

High 1440 x 1152 1440 x 1152 4:2:21440 60 Mb/s 60 Mb/s 1440 x 1152

80 Mb/s

Main

4:2:0 4:2:0 4:2:2 4:2:0 4:2:0 or720 x 576 720 x 576 720 x 608 720 x 576 4:2:260 Mb/s 15 Mb/s 50 Mb/s 15 Mb/s 720 x 576NO B 20 Mb/s

Low

4:2:0 4:2:0352 x 288 352 x 2884 Mb/s 4 Mb

Table A.1.: Profiles and levels in MPEG-2 [16]

A. Appendix 89

A.3. Simulink Models

The next figures present the Simulink blocks used to simulate the video adaptationsystem.

ZOH3

ZOH2

ZOH1

Uniform RandomNumber

Uniform RandomNumber

RepeatingSequence

QAFSModel

ANB '

RNB

Epsilon

ANB*

PID-Controllerwith antiwindup

E act_

IN

act_

OU

T

AN

B

RN

B

U

Delay

z

1

Constant

2E5Buffer

BandwidthSensor

In 1

In 2Out1

Buffer

Fill out Rate

Fill in Rate

Buffer Level

ANB

ANB

FeedbackFeedback

error

act_IN

act_IN Fill in Rate

RNB

Phi

Figure A.2.: System overview

ANB*

1

Unity

1

Unit Delay

z

1

Switch

Subtract

Relational

Operator

<

ProductEpsilon

3

RNB

2

ANB '

1

QAFS

coefficient

ABW '

RBW

Figure A.3.: QAFS frame adaptation model

A. Appendix 90

U

1

PID

controller

Error

Antiwindup

Output

State

Antiwindup

State

ANB

RNB

act_IN

act_OUT

State_out

RNB

5

ANB

4

act_OUT

3

act_IN

2

E

1

Figure A.4.: The controller connection

State

2

Output

1

Zero-Order

Hold3

Zero-Order

Hold2

Zero-Order

Hold1

Gain

c3

c3

Gain

c2

c2

Gain

c1

c1

Delay-

Computation

Time4

z

1

Delay-

Computation

Time3

z

1

Delay-

Computation

Time

z

1

Antiwindup

2

Error

1E(k)

E(k-1)

E(k-2)

Figure A.5.: PID algorithm

State_out

1

ZOH2

ZOH1

Embedded

MATLAB Function

statevar

act_in

act_out

anb

rnb

awu

state_out antiwindup

Delay-

Computation

Time

z

1

Anti-windup

coefficient

awu

act_OUT

5

act_IN

4

RNB

3

ANB

2

State

1

Figure A.6.: Antiwindup modelThe code used in “Embedded Matlab Funcion” is shown in Section 4.3.4 Code 1.

A. Appendix 91

Buffer

Level

1

ZOH

Subtract

Discrete-Time

Integrator

1

z-1

Compare

To Zero

< 0

Fill in

Rate

2Fill out

Rate

1

Figure A.7.: The buffer model

Out1

1

Convert

Data into

Bandwidth

1/ ts

Buffer Level

Sensor

Buffer Contents

Positive Input 1

Buffer Before Flush

In 2

2

In 1

1

Sensor Out

Figure A.8.: Sensor overview

Buffer

Before

Flush

1

ZOH

Unit Delay

z

1SubtractSaturation

Convert

BW into

Data

ts

Positive

Input 1

2

Buffer

Contents

1

Figure A.9.: Buffer level sensor

Bibliography

[1] Reinder J. Bril. A QoS approach for multimedia consumer terminals withmedia processing in software, 2007.

[2] Giorgio C. Buttazzo. Hard real-time computing systems: predictable

schedulling and applications. Springer, second edition, 2005.

[3] T. Cucinotta, L. Palopoli, and L. Marzario. Stochastic feedback-based controlof qos in soft real-time systems, 2004.

[4] Richard C. Dorf and Robert H. Bishop. Modern control systems. PearsonPretince Hall, tenth edition, 2005.

[5] Gene F. Franklin, J. David Powell, and Abbas Emami-Naeini. Feedback control

of dynamic systems. Pearson Pretince Hall, fifth edition, 2006.

[6] Gene F. Franklin, J. David Powell, and Michael Workman. Digital Control of

Dynamic Systems. Addison-Wesley, third edition, 1998.

[7] Ningning Hu and Peter Steenkiste. Evaluation and characterization of avail-ablebandwidth probing techniques. 21, August.

[8] Clemens Christian Wust Isovic. Inteligent Control for Scalable Video Process-

ing. PhD thesis, Eindhoven Universty of Technology, November 2006.

[9] Damir Isovic. Flexible Scheduling for Media Processing in Resource Con-

strained Real-Time Systems. PhD thesis, Malardalen, Sweden, November.

[10] M. Jain and C. Dovrolis. End-to-end available bandwidth: measurementmethodology, dynamics, and relation with tcp throughput, 2002.

[11] Anand K.V.M.S. and Pradeep H.R. Online stream adaptation under con-strained network bandwidth. Master’s thesis, Technische Universitat Kaiser-slautern, International Institute of Inforamtion Technology Bangalore, June2007.

[12] B. Li and K. Nahrstedt. A control-based middleware framework for qualityof service adaptations. IEEE J. Select. Areas Commun., September 1999.

92

Bibliography 93

[13] J. Strauss, D. Katabi, and F. Kaashoek. A measurement study of availablebandwidth estimation tools, 2003.

[14] Andrew S. Tanenbaum. Computer Networks. Elsevier, forth edition, 2003.

[15] Zhou Wang, Hamid R. Sheikh, and Alan C. Bovik. The Handbook of Video

Databases: Design and Applications, chapter Chapter 41, pages 1041–1078.CRC Press, 2003.

[16] John Watkinson. The MPEG Handbook. Elsevier, second edition, 2005.

Errata

In page 36 Figure 4.2, and page 89 Figure A.2 must be:

Figure E-1: System overview

In page 51 Figure 4.7, and page 89 Figure A.2 must be:

Figure E-2: QAFS model used in simulation

1

Documents

Control Theory Approaches to Resource …rts.eit.uni-kl.de/fileadmin/publication_files/Coelho...Control Theory Approaches to Resource Adaptation in Real Time Embedded Systems Master