IMAGE SCALING ALGORITHMSBy SUMEET SAURAV

INTRODUCTION: THE INTERPOLATION PROBLEM

Assume a unit square between the four nearest pixels.

The target pixel will then be some x fraction and y fraction into this unit square.

Interpolation problem then consists of finding suitable values for these fractions.

NO NEW DATA will be added by ANY interpolation scheme! The best you can do is minimize interpolation artifacts.

CATEGORIES OF INTERPOLATION METHOD

Interpolation is done by two methods: By interpolating the pixels based on the characteristics of

local features, such as edge information or neighboring pixel information.

Based on these differences interpolation method can be classified into two categories:

A)Adaptive

B)Non-adaptive

Non-adaptive is suitable for real time hardware implementation.

TYPES OF INTERPOLATION

Nearest Neighbor Bilinear Interpolation. Bicubic convolution Interpolation. Cubic convolution Interpolation Extended Linear Interpolation Win Scale Area Average Lanczos resampling Hqx(Pixel art scaling algorithm) Super scaling Vectorization

NEAREST NEIGHBOR INTERPOLATION Every pixel value in the output product is set

to the nearest input pixel value.Pros:1)Very simple and fast.2) No new values are calculated by interpolationCons:1) Some pixels get lost and others are

duplicated.2) Loss of sharpness.

BILINEAR INTERPOLATION Calculation of the new pixel value is performed by the

weight of the four surrounding pixels.

Pros:

1) Extremas are balanced

2)Image losses sharpness compared to Nearest Neighbor.

Cons:

1) Less contrast compared to Nearest Neighbor.

2) New values are calculated which are not present in the input product.

CUBIC CONVOLUTION Calculation of the new pixel value is performed by

weighting the 16 surrounding pixels.

Pros:

1. Extremas are balanced.

2. Image is sharper compared to Bi-linear Interpolation

Cons:

3. Less contrast compared to Nearest Neighbor.

4. New values are calculated which are not present in the input product

5. Slow, compared to Nearest Neighbor resampling

WINSCALE:AN IMAGE SCALING ALGORITHM

The winscale algorithm uses an area pixel model and a domain filtering method for image scaling.

The algorithm uses an area-model rather than a point-model for a pixel evaluating.

The pixel shapes of the original and the scaled image are treated as rectangles and pixel intensity is evenly distributed in rectangle area.

New pixel of the scaled image is obtained by weighted-averaging the pixel values of the original pixels with area coverage ratio.

A source image is the original image before scaling up/down. Target image refers to the scaled image. The region of the target pixel being calculated currently is

called a filter window.

IMAGE UP-DOWN SCALE

If the filter window is perfectly included in one source pixel region, sampling has the nearest neighbor property: the pixel value becomes equal to that of source pixel.

This property makes good high frequency response and reduces blurring effect.

If the filter window is spread over multiple source pixels, sampling has the linear property like bilinear: smoothness is added.

Therefore, winscale inherits two good properties of both nearest and linear: good high frequency response and good smoothness.

EDGE ORIENTED IMAGE SCALING Efficient area-pixel scaling technique is used. A simple edge catching technique is adopted to preserve

the image edge features.

When up-scaled the area of the target pixel is less than that of the source pixel.

A window is put on the current target pixel to calculate its estimated luminance value.

The number of source pixels overlapped by the current target pixel window is one, two, or a maximum of four.

Let the luminance values of the source pixels overlapped by the window of the current target pixel at (k,l) is given by

The estimated value of the current target pixel is given by

The areas of the overlapped regions can be calculated as:

Where

[A(m,n),A(m 1,n),A(m,n 1),A(m 1,n 1)]

[ ( , ) top(k, l), right(k, l) top(k, l), left(k, l) bottom(k,l),

right(k, l) bottom(k,l)]

left k l

( , ) (m,n)

top(k, l) heightA(m,n)

right(k, l) widthA(m 1,n 1)

bottom(k,l) height(m 1,n 1)

left k l widthA

The area-pixel scaling algorithm requires total of 13 addition,8 multiplications and one division to calculate one target pixel.

The proposed low-complexity algorithm uses the following set of equations:

where

[A'(m,n),A'(m 1,n),A'(m,n 1),A'(m 1,n 1)]

[left'(k, l) top'(k, l), right'(k, l) top'(k, l),

left'(k, l) bottom'(k, l), right'(k, l) bottom'(k, l)]

[ '( , ), top'(k, l), right'(k, l), bottom'(k, l)]

Appr[left(k, l), top(k, l), right(k, l), bottom(k,l)]

left k l

Here a low-cost edge-catching technique is employed to preserve the edge features to obtain better visual quality.

The final areas of the overlapped regions are given as

Where the tuning operator is used to tune the areas of four overlapped regions according to the edge features obtained by edge-catching technique.

[ ''( , ), ''( 1), ''( , 1), ''(m 1,n 1)]

([A'(m,n),A'(m 1,n),A'(m,n 1),A'(m 1,n 1)]

A m n A m A m n A

Assuming the width and height of the target pixel window are denoted as win(w) and win(h)

Here win(h) and win(w) are determined based on current magnification factors (mf_w for x-direction and mf_h for y-direction)

The value of mf_w and mf_h is given by

( ) ( )Asum win w win h

_ /

_ /

mf w TW SW

mf h TH SH

'( , ) ( ) '( , )

'( , ) ( ) '( , )

right k l win w lefr k l

bottom k l win h top k l

The value of left’(k,l) is determined from the fig

'( , ) min(SRCright(m,n) ( , ), ( ))left k l WINleft k l win w

DETERMINATION OF AND

Initially we have

Both Sw and Sh are rounded to integers.

(0,0) ( ) / 2

SRCright(0,0) Sw

WINtop(0,0) ( ) / 2

WINleft Sw WINw

Sh WINh

SRCbtm Sh

w h

OVERLAPPING METHOD FOR ROUNDING DOWN ERROR Let represent the regulating times

required for each row, thus it can be given asrw

OVERLAPPING METHOD FOR ROUNDING UP ERROR

THE LOW-COST EDGE-CATCHING TECHNIQUE Sigmoidal signal is taken as image-edge

model for image scaling. E(k) represents the luminance value of the

pixel at coordinate k. e

An Image-Edge Model Local Characteristics

The estimated value of the pixel to be interpolated is determined by using linear interpolation.

The interpolated value might not match greatly with E(k).

To solve this problem the distance s is modified to make approach E(k) for better estimation.

Here an evaluating parameter L is defined to estimate the local characteristics of the data in the neighborhood of k. It is given as

L=0 indicates symmetry, so s is unchanged. L>0 indicates that variation between E(m+1)

and E(m-1) is quicker than between E(m+2) and E(m).

It means that edge is more homogeneous to the right side.

L<0 indicates the edge is more homogeneous on the left side so s must be decreased to obtain a better estimation

The modified equation for estimating the interpolated pixel value is given by

Where s’ is calculated with the following equation

The same concept of 1-D edge-catching technique can be used in the proposed 2-D scaling processor.

Let be the evaluating parameter to estimate the local characteristics of the current pixel at the coordinate(k,l).

If top’(k,l) is greater than or equal to win(h)/2 it means that A’(m,n) is bigger than or equal to A’(m,n+1).

Hence the upper row(n) is more important than the lower row(n+1) to catch edge features.

Thus is given by

If =0 indicates symmetry, so A’(m,n) is unchanged.

If >0 indicates that the variation between E(m+1,n) and E(m-1,n) is quicker than between E(m+2,n) and E(m,n).

It means that edge is more homogeneous on the right hand side so we increase A’(m+1,n).

If <0 the edge is more homogeneous on the left-hand side ,thus we decrease A’(m+1,n) to obtain a better estimate.

Based on the above rules the final areas of the overlapped region is given by:

Where Ac=A’(m,n) if and Ac=A’(m+1,n) if If top’(k,l) is less than win(h)/2 ,it means that

A’(m,n) is smaller than A’(m,n+1).Hence the lower row(n+1) is more important than the upper row(n) to catch the edge features.

Thus is given by

0LA0LA

The final area of the overlapped regions are given by

Where Ac=A’(m,n+1) if and Ac=A’(m+1,n+1) if

0LA

0LA

VLSI ARCHITECTURE

Approximate module

Edge catcher Area

generator

Area tuner

Target generato

r

ARCHITECTURE OF REGISTER BANK

ARCHITECTURE OF AREA GENERATOR

ARCHITECTURE OF EDGE CATCHER

ARCHITECTURE OF AREA TUNER

ARCHITECTURE OF TARGET GENERATOR

AREA-AVERAGE ALGORITHM The algorithm can be done on horizontal and vertical

dimensions separately with minimal correlation which helps to exploit the parallel capabilities of the FPGA.

The scaling algorithm can be performed stepwise and without causing image quality loss.

If we need to reduce a image f(assume f =a/b>1) times, first we magnify each pixel in the original image b times and then shrink the enlarged image a times.

ALGORITHM IMPLEMENTATION

RESOURCE AND SYSTEM PERFORMANCE COMPARISON

BI-CUBIC INTERPOLATION ALGORITHM This algorithm requires 16 source pixels for getting the

interpolated value of the target pixel. It preserves fine details of the image .

Where

32

2

2

( )

1 (1 ) (1 )

2 (1 2 )

3 (1 )

4 (1 )

l round x a x l

t a a a

t a a

t a a a

t a a

32

2

2

( ) b=y-k

t5=-b (1-b) (1-b)*t1

t6=(1-2 )

7 *(1 )

8 *( 1)

k round y

b b

t b b b

t b b

HARDWARE ARCHITECTURE FOR BI-CUBIC INTERPOLATION

The main components of the architecture are: (1) Input Buffer Bank (2) Output Buffer Bank (3) Off-Chip RAM (4) Parallel Zoom

Processors (5) Control Unit (6) Interpolation Coefficient

Generator

The main components of the Zoom Processor are:

(1) Pixel demultiplexer (2) Pixel Registers (3) Coefficient Demultiplexer (4) Coefficient Registers (5) Temporal Registers (6) Output Pixel Register

ZOOM PROCESSOR ARCHITECTURE

THE CONTROL UNIT The unit synchronizes external memory,

input and output buffers bank, zoom processors and interpolation coefficient generator.

For each clock cycle, the Control Unit generates t1, t2, t3 and t4 (vertical coefficient) and t5, t6, t7 and t8 (Horizontal coefficients).

In the cycle L+4, the Zoom Processor 1 captures t1, t2, t3 and t4 and in cycle L+5 output the first result.

The control unit indicates which processor reads the Horizontal coefficients and when all the Zoom Processors has given one result.

PIPELINE SCHEME FOR ZOOM PROCESSOR

INTERPOLATION COEFFICIENT GENERATOR

The Interpolation Coefficient Generator generates dynamically the 4 by 4 input mask that will be used by the Zoom processors to obtain the output pixels.

The Interpolation Coefficient Generator gives a result each clock cycle after 5 latency cycles.

LINEAR SPLINE INTERPOLATION

Given , fit linear splines to the data.

(X 0, Y 0),(X1, Y1),(X 3, Y 3)..........(Xn, Yn)

),()()(

)()( 001

010 xx

xx

xfxfxfxf

),()()(

)( 112

121 xx

xx

xfxfxf

),()()(

)( 11

11

nnn

nnn xx

xx

xfxfxf

10 xxx

21 xxx

nn xxx 1

QUADRATIC SPLINE In these splines, a quadratic polynomial

approximates the data between two consecutive data points.

Given fit quadratic splines through the data.

nnnn yxyxyxyx ,,,,......,,,, 111100

,)( 112

1 cxbxaxf ,22

22 cxbxa

,2nnn cxbxa

10 xxx

21 xxx

nn xxx 1

TWO-POINT CUBIC SPLINE INTERPOLATION The most widely used scaling algorithms in usually

occasion are nearest neighbor, bilinear, and bicubic. Nearest neighbor and bilinear interpolation are simplest

and fastest, but the images come from them are either blocky or over-smoothed.

The bicubic can get a better image quality than them but it is more complicated because it needs a 4x4 neighbor.

The nearest neighbor has good high frequency response and can preserved more edge information of the original image. But it degrades image quality due to aliasing.

The weakness of bilinear and bicubic is blur effect causing bad high frequency response.

The proposed algorithm has a good high frequency response approximated to nearest neighbor and it is as simple and fast as the bilinear.

It can get an image quality as good as the bicubic.

Interpolation Kernel Curve

EXTENDED |LINEAR INTERPOLATION The kernel of the linear interpolation is built

up of first-order polynomials and the weighting coefficient is given by the relation:

The kernel of the cubic convolution interpolation is built up of third-order polynomials whose weighting coefficient is given by

KEYS’ AND BI-CUBIC WEIGHTING COEFFICIENT. Key determined the value of c=-0.5.

Bi-cubic interpolation uses the constant c=-1 and the weighting coefficient of the bi-cubic interpolation is given by

DECOMPOSITION OF THE 2-D BI-CUBIC

THE KERNEL OF EXTENDED LINEAR INTERPOLATION It has low complexity as well as it improves

the quality of the linear interpolation. Here the number of sampling points of the

third order polynomial interpolation is used. The weighting coefficients are given by

DERIVATION OF EL KERNEL FROM BI-CUBIC

Using the coefficient of the Bi-cubic weighting coefficient we have

After certain manipulation the above equation can be modified and the required weighting coefficient is given by

Where is called the sharpness factor.

SHARPNESS FACTOR DECISION

COORDINATE ORIENTATION AND THE INTERVAL CALCULATION Here the origin of a source pixel in source

coordinates is (0.5,0.5),although its location in the memory is at(1,1).

The coordinate of the interpolated pixel in the x-direction and y-direction is

The vertical coordinate of the interpolated pixel(Ym) gives the row address of the source pixel which is determined as follows:

Thereafter the address neighboring pixels are determined as follows:

Similarly column address can be calculated by using Xm of the interpolated pixel.

The distance in vertical and horizontal direction is given by:

REDUCTION OF THE COMPUTATION OF THE WEIGHTING COEFFICIENT

Vertical Weighting Coefficients

Horizontal Weighting Coefficients

DECOMPOSITION OF 2-D 4X4 INTERPOLATION

Vertical interpolation is performed first and four virtual pixels are calculated using four 1-D interpolation. The generated virtual pixel is given by:

Now from the vertically interpolated virtual pixels actual interpolated pixel is obtained by horizontal interpolation using one 1-D horizontal interpolation. The interpolated pixel is given by:

All the interpolated points within the same row of Q have the same vertical weighting coefficients.

These same vertical weighting coefficients are calculated only one time for each row and then proceeds the calculations of horizontal weighting coefficients of each interpolated pixel in that row.

The number of memory access time is given by

THE PROPOSED HARDWARE

THE COORDINATE CALCULATION UNIT

THE WEIGHTING COEFFICIENT GENERATOR

The most computation effort in extended linear interpolation is the calculation of interpolation weighting coefficients.

THE VERTICAL AND HORIZONTAL INTERPOLATION UNIT

THE VIRTUAL PIXEL BUFFER The scale ration determine both input and output data

rates of this buffer. While scaling up the data rate of vertical interpolator is

less than that of the horizontal interpolator in a period of time.

During scaling down the converse case happens.