Victor%27s Video Scaler Cook Book

  • View
    361

  • Download
    4

Embed Size (px)

Text of Victor%27s Video Scaler Cook Book

Video Scaler CookBook

Victor Ramamoorthy Software Engineering Group Information Appliance National Semiconductor Corp Santa Clara, CA 30 October 2002

________________________________________________________________________ Video Scaler CookBook Victor Ramamoorthy Page: 1 (30) 10/13/2004

Contents Video Scaler CookBook ..................................................................................................... 1 Video Scaler Cook Book .................................................................................................... 3 Introduction..................................................................................................................... 3 Performance of a Video Scaler ....................................................................................... 4 Scaling Fundamentals ..................................................................................................... 6 Example ...................................................................................................................... 7 Linear Interpolation .................................................................................................... 8 Choosing a better predictor............................................................................................. 9 Mirroring Pixels ............................................................................................................ 11 Implementing the Interpolating Filter........................................................................... 11 Putting It All Together .................................................................................................. 14 Finally ........................................................................................................................... 15 Tables............................................................................................................................ 17

________________________________________________________________________ Video Scaler CookBook Victor Ramamoorthy Page: 2 (30) 10/13/2004

Video Scaler Cook Book

IntroductionVideo Scaler is a device that stretches or shrinks images or video frames. For the purposes of illustration, we have a source image as shown below:

Source Image When expand the size of the image, the total number of pixels contained in the destination image is more than what is available in the source image. Hence a whole lot of new pixels have to be created by the scaler. When we shrink the image, we may keep a part of the pixels and throw the rest. In addition, we may still have to generate new pixels even if we are shrinking the image. The scalers job is simple: (1) It decides what pixels to keep in the source image and (2) what pixels have to be recreated. In this cookbook we will explore the scalers anatomy.

Destination Image: Shrinking Note that a scaler can stretch an image only in X direction or only in Y direction or both. We want to design a scaler that can achieve arbitrary1 image sizes without noticeable quality loss. There is no restriction to keep the aspect ratio2 as constant while scaling.

1 2

Arbitrary sizes, memory size permitting, filter lengths conducive. The ratio of width and height of an image is called the aspect ratio.

________________________________________________________________________ Video Scaler CookBook Victor Ramamoorthy Page: 3 (30) 10/13/2004

Destination Image: Expansion No distinction is made between shrinking and expansion of an image as far as the scaler algorithm is concerned. They are the opposite sides of the same problem. The scaler will operate exactly the same way if we are expanding or squeezing the source image. This cookbook describes the process with which smooth expansion/contraction scaling is obtained. It also details the construction of a hardware system, which can operate at video frame rates. In particular, the focus of the treatment is to spell out the different parameters that control the performance of the scaler. This cookbook is aimed at engineers who are not familiar with video scaling problem. Experts on the subject are warned that they may be wasting their time reading this cookbook. Many of the theoretical issues are merely glossed over and the attention is centered on the implementation. Useful tables at the end contain filter coefficients that can be used in a practical situation.

Performance of a Video ScalerAs with any engineering design, we have to ask critical questions like what are we designing and how do we know it is good. When it comes to video scaler, that kind of questions are tough to answer because the quality of the end result simply cannot be measured. Video quality is a subjective thing. It varies with the content and viewing conditions. More importantly, when we scale video frames, there is no reference to compare. We are creating a new frame that was not there before, and we have to compare it with the original frame with a different size. It is like comparing apples and oranges. The best way to get around this problem is to ask if the scaled frame looks like the original and hunt for artifacts that might have crept in the scaling process. Still we are looking for something that has no reference and has some how appeared on the end result which is not a good way to measure performance. So we come back and settle down to the measurement of how it looks after scaling.

________________________________________________________________________ Video Scaler CookBook Victor Ramamoorthy Page: 4 (30) 10/13/2004

The primary performance measure is undoubtedly the visual quality of the scaler. In general, visual quality is not easily measured by metrics such as Picture Signal-to-Noise Ratio and Mean Square Error, though they are often quoted in the literature. The secondary performance measures are Computational Load or Complexity of Design. We will attempt to quantify these measures wherever meaningful. Another thing to remember is that performance is a function of color conversion stages employed. For instance, if the source image is RGB, see fig. 1 below, the scaling can be done in the RGB domain. Notice that scaling is done on the component images. If the image is only available as YUV domain then scaling can be done in YUV domain as in fig.2: R Source Image G B Fig.1 Y Source Image U V Fig.2 R Source Image (RGB) Fig. 3 As suggested by fig.3 above, the color conversion can be included in the last stage before destination image is displayed. Or it also can be done before scaling as shown below: Y Source Image (RGB) Fig.4 Color Conversion U V Scaler Scaler Scaler Y U V Destination Image (YUV) G B Scaler Scaler Scaler R G B Color Conversion Destination Image (YUV) Scaler Scaler Scaler Y U V Destination Image Scaler Scaler Scaler R G B Destination Image

________________________________________________________________________ Video Scaler CookBook Victor Ramamoorthy Page: 5 (30) 10/13/2004

The performance of the system in fig.3 and fig.4 will in general be different. Because of additional operations involved with color conversion, there is ample opportunity to inject additional noise in the system. However, by careful choice, we can make the visual qualities of these two systems to approach each other. Another important thing to note is that the scaler operates on Planar images. A planar image could be an image containing just red color, or Y values. It is a component image. Repeating the scaling on all other component images does complete scaling.

Scaling FundamentalsLet

x ,ys

s

be the size in the X direction and Y direction respectively of a planar source

image. Let the corresponding sizes of the destination image be the source image has

x ,yd

d

. This means that

x

s

pixels in a row, which must be converted to =

x

d

pixels in the

destination image. The ratio of these two numbers is called scaling factor

S

x

x x

d s

.

(1)

Similarly the scaling factor in the Y direction is

S

y

=

y y

d s

. Note that the scaling factor

is the ratio of two integers and is hence a real (floating-point) number. We will just illustrate the method for scaling in the X direction in this cookbook as the same kind of operations can be duplicated for Y direction. The key idea here is to expand the source size range to include floating point numbers. S What is the size range? Well, it is the set of integers I x ={0, 1, 2, 3, xs -1} which

denote the pixel positions3 in the X direction of the source image. The first pixel position starts at 0. The last one ends at xs -1. We can expand this range to contain all real numbers not just integers. Note that this is our mental construct and not a real thing. When everything is finished we will map this back into integers, as a pixel position S cannot be a fractional number. Let us denote this new range as R x containing all numbers from 0 to

x

s

-1.

Now we are ready to map the destination pixel positions back onto the source pixel S positions, i.e., mapping onto the range R x . Why is this needed? We have the job of generating new pixels in the destination image and they must have the resemblance of the source image. We need to know where to place these new pixels. We also need to knowWe are talking about pixel positions which are sequential numbers starting at 0 from left. Do not confuse this with the value of a pixel. We will use the notation V(.) to denote the value of a pixel.3

________________________________________________________________________ Video Scaler CookBook Victor Ramamoorthy Page: 6 (30) 10/13/2004

their values.