876 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 6, …moncef/publications/vector-rational-interpolation.pdf · supported by the European RTN Project RTN1–1999-00177 ... of color image

876 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 6, NO. 6, DECEMBER 2004

Vector Rational Interpolation Schemesfor Erroneous Motion Field EstimationApplied to MPEG-2 Error Concealment

Sofia Tsekeridou, Member, IEEE, Faouzi Alaya Cheikh, Moncef Gabbouj, Senior Member, IEEE, andIoannis Pitas, Senior Member, IEEE

Abstract—A study on the use of vector rational interpolation forthe estimation of erroneously received motion fields of MPEG-2predictively coded frames is undertaken in this paper, aimingfurther at error concealment (EC). Various rational interpolationschemes have been investigated, some of which are applied todifferent interpolation directions. One scheme additionally usesthe boundary matching error and another one attempts to locatethe direction of minimal/maximal change in the local motion fieldneighborhood. Another one further adopts bilinear interpolationprinciples, whereas a last one additionally exploits available codingmode information. The methods present temporal EC methods forpredictively coded frames or frames for which motion informationpre-exists in the video bitstream. Their main advantages are theircapability to adapt their behavior with respect to neighboring mo-tion information, by switching from linear to nonlinear behavior,and their real-time implementation capabilities, enabling them forreal-time decoding applications. They are easily embedded in thedecoder model to achieve concealment along with decoding andavoid post-processing delays. Their performance proves to be sat-isfactory for packet error rates up to 2% and for video sequenceswith different content and motion characteristics and surpass thatof other state-of-the-art temporal concealment methods that alsoattempt to estimate unavailable motion information and performconcealment afterwards.

Index Terms—Error concealment, motion field estimation,MPEG-2 compression, temporal concealment, transmission er-rors, vector rational interpolation.

I. INTRODUCTION

TRANSMISSION of highly compressed video bitstreams(e.g., MPEG-2 video bitstreams) through physical com-

munication channels is liable to errors, being either packet/cellloss or bit errors (isolated or in bursts). These types of errors areusually referred to as bitstream errors and they frequently resultin loss of the decoder synchronization, which becomes unableto decode until the next resynchronization codeword in the re-ceived bitstream. Since resynchronization points are placed at

Manuscript received October 3, 2001; revised January 7, 2003. This work wassupported by the European RTN Project RTN1–1999-00177 – MOUMIR. Theassociate editor coordinating the review of this manuscript and approving it forpublication was Dr. Heather Hong.

S. Tsekeridou is with the Electrical and Computer Engineering Depart-ment, Democritus University of Thrace, 67100 Xanthi, Greece (e-mail:[email protected]).

F. A. Cheikh and M. Gabbouj are with the Institute of Signal Processing,Tampere University of Technology, FIN-33101 Tampere, Finland (e-mail:[email protected]; [email protected]).

I. Pitas is with the Department of Informatics, University of Thessaloniki,Thessaloniki 54124, Greece (e-mail: [email protected]).

Digital Object Identifier 10.1109/TMM.2004.837266

the header of each slice by an MPEG-2 coder, unless otherwisespecified, transmission errors may lead to partial or entire sliceinformation loss (loss of prediction errors, coding modes, mo-tion vectors). Such loss of synchronization leads to observabledeterioration of the decoded sequence quality, which, in the caseof MPEG-2 coding, is further attributed to the use of VLC anddifferential coding. Furthermore, due to the use of motion-com-pensated prediction, temporal error propagation to predictivelycoded frames inside a group of pictures (GOP), i.e., propaga-tion errors, leads to even worse results.

A number of methods have been proposed in the literature toachieve error resilience and thus deal with the above-mentionedproblem. A nice overview of such methods is given in [1]. Oneof the ways to solve this problem is the implementation of errorconcealment (EC) methods at the decoder side. Such methodsexploit the spatial and/or temporal correlations inside andamong correctly received neighboring regions/frames [2]–[12].A subclass of EC methods focuses on the “optimal” estimationof erroneously received motion fields based on available sur-rounding information. Among the methods belonging in thissubclass, one may distinguish:

• the zero motion EC (ZM EC), which sets lost motion vec-tors to zero and, thus, performs temporal replacement;

• the motion-compensated EC (MC-AV or MC-VM EC),which calculates the average or vector median of adjacentpredictively coded motion vectors;

• the boundary matching algorithm EC (BMA EC), whichdefines a number of motion vector candidates and selectsthe optimal one that minimizes the boundary matchingerror [13];

• the motion vector estimation by boundary optimizing EC(MVE-BO EC), a fast suboptimal version which firstlyevaluates the vector median (a MAP motion estimate) ofadjacent predictively coded motion vectors, then definessearch regions in reference frames around the blockpointed at by the vector median, and finally looks forthe optimal motion vector by minimizing the boundarymatching error [2];

• the forward-backward block matching EC (F-B BM EC),which performs temporal block matching of adjacentblocks in order to estimate their “best match” in thereference frames by MAD minimization [10];

• the decoder motion-vector estimation EC (DMVE EC),a variant of F-B BM EC which, however, uses smaller

1520-9210/04$20.00 © 2004 IEEE

TSEKERIDOU et al.: VECTOR RATIONAL INTERPOLATION SCHEMES FOR ERRONEOUS MOTION FIELD ESTIMATION 877

neighboring regions in the temporal matching process toreduce processing delays [11];

• the bilinear motion field interpolation EC (BMFI EC),which estimates one motion vector per pixel instead ofblock using bilinear interpolation among adjacent motionvectors [12];

• the combined BMFI EC (Comb. BMFI EC), an extensionof the BMFI EC which, additionally, exploits the outcomeof the BMA EC [12].

Motivated by the introduction of vector rational interpola-tion in [14]–[16] and its remarkable performance in the caseof color image interpolation/resampling, a study on its use forthe estimation of erroneously received motion fields of predic-tively coded frames inside an MPEG-2 compressed video bit-stream is undertaken in this paper. It is evident that the pre-sented methods may be used alongside any hybrid coder thatuses predictive coding. A number of different motion vector ra-tional interpolation schemes are investigated in this paper forEC purposes. Initially, four distinct interpolation schemes arepresented differing in which pairs of adjacent motion vectorsare used in the vector rational interpolation function, i.e., to-ward which direction the interpolation takes place. Then, theadditional concept of boundary matching error minimization isincorporated in the presented approaches to select the optimalestimation out of the four estimated motion vector candidates.Another scheme attempts to locate the direction of minimumchange in the local motion field neighborhood, based on adja-cent information, and to perform vector rational interpolationin that direction. Another variant has introduced concepts sim-ilar to the ones presented in [12] by including bilinear inter-polation principles to the motion vector rational interpolationapproach and by introducing a finer interpolation grid in orderto further exploit existing spatial correlations. Finally, existinginformation about the coding modes of adjacent macroblocks(MBs) is exploited and motion vector rational interpolation isemployed only among the available inter-coded neighbors. Theperformance of all presented schemes is compared against thatof the motion-compensated temporal ECs mentioned above byjudging from the achieved visual quality of the concealed videosequences and by comparing the processing time required forconcealment and the accuracy of the restored motion field. Suchconcealment methods perform satisfactorily in cases of low- tomedium-valued packet error rates (PER). For higher PERs, asatisfactory solution is the combination of layered coding withEC methods [7], [17].

The organization of the paper is as follows. In Section II,the mathematical definition and the principles of vector rationalinterpolation are briefly summarized. In Section III, a variety ofmotion vector rational interpolation methods are presented forEC of predictively coded frames. Simulation results are reportedin Section IV and conclusions are drawn in Section V.

II. VECTOR RATIONAL INTERPOLATION

Vector rational interpolation has been introduced in [14]–[16]for color image interpolation/resampling applications, whereevery pixel is represented as a 3-component vector in a colorspace, in order to exploit the inherent correlations among

different color components. The input/output relationship ofa vector rational function (VRF), according to the definitiongiven in [14], is given by

(1)

where are -component input vectors to thefilter, is a vector-valued polynomial, a scalar one andthe th component of the VRF output given by

(2)

where

(3)

(4)

denotes the vector norm, denotes integer part, thecoefficients , are constant real numbers, whereas thecoefficients are expressed as a nonlinear function ofthe input vectors

(5)

If , (1) reduces to the scalar rational function input/outputrelationship.

III. MOTION VECTOR RATIONAL INTERPOLATION EC SCHEMES

In this section, we investigate the use of properly definedvector rational interpolation functions in the estimation processof erroneously received motion fields of predictively codedvideo frames. Information loss due to transmission errors inMPEG-2 compressed video data translates to missing motionvectors, coding modes and prediction errors of the respectiveMBs. If default synchronization is used by the MPEG-2 en-coder, i.e., resynchronization points exist at the start of eachslice, which is an entire row of MBs, only correctly received topand bottom adjacent block data is available for the estimationof the lost one. In subsequent subsections, a number of differentmotion vector rational interpolation approaches are introduced.The aim of all presented schemes is to estimate the lost motionvector(s) in the best possible way by using information fromavailable adjacent motion vectors. The neighborhood consid-ered in all interpolation schemes is shown in Fig. 1. After thelost motion vector has been estimated in the way presentedin subsequent subsections, concealment of predictively codedframes is performed by copying the displaced block of thepreviously decoded frame with respect to the estimated motionvector to the current lost one. In the case of B-frames, wheretwo motion fields are available (forward and backward), lostmotion vector estimation is accomplished in both fields and


Fig. 1. Neighborhood employed in the motion field interpolation approaches.

Fig. 2. MVRI schemes: (a) 2-stage 1-D case; (b) 2-D case; (c) 2-stagecombined 1-D and 2-D case; and (d) 2-D case considering all directions.

the estimate that leads to the minimum boundary matchingerror (refer below for definition) is selected for concealment.Intra-coded frames are concealed by the F-B BM EC [10].Motion vector rational interpolation can also be employed forrecovering lost concealment motion vectors of I-frames whensuch are generated and transmitted.

A. Introduction of Four Distinct Approaches

In a first approach, four different interpolation schemes havebeen considered [18], differing only in the neighboring pairsof available motion vectors used, that is, in which direction theinterpolation is applied, as is shown in Fig. 2. They are describedin the sequel:

2-Stage 1 – D Case [Fig. 2(a)]:Horizontal Interpolation: Initially, estimates ( , ) of

the top and bottom adjacent motion information are obtained byapplying the 1-D vector rational interpolation function

(6)

(7)

where , and coeffi-cients , , , are defined by

(8)

In (8), denotes the vector norm and is a positive con-stant that controls the degree of nonlinearity of the rational filter.

Vertical Interpolation of Horizontal Estimates: The lostmotion vector is finally estimated by averaging the horizontalestimates and

(9)

2-D Case [Fig. 2(b)]: This interpolation scheme estimatesthe lost motion vector by employing the expression

(10)

where and is given by (8).2-Stage Combined 1-D and 2-D Case [Fig. 2(c)]:

Horizontal Interpolation: It is performed in the same wayas in the 1-D case.

Combined Interpolation: The horizontal estimates of theprevious stage are used as input motion vectors along with theinput motion vectors of the 2-D case to estimate the lost motionvector

(11)

where and is againgiven by (8).

2-D Case of All Directions [Fig. 2(d)]: This interpolationscheme is an extension of the 2-D case, in the sense that almostall directions between neighbors are considered in the final es-timation

(12)

where now

All interpolation schemes attempt to estimate lost motion in-formation in such a way that motion field smoothness is attainedin smooth motion areas, while at the same time irregular mo-tion of adjacent blocks does not result in high estimation errors.Intra-coded neighbors are simply considered here as having zerovalued motion vectors. No coding mode information of adja-cent blocks is exploited at this point. In smooth motion areas,where Euclidean distances between adjacent vectors are small,the weights are close to 1.0, thus leading to an averaginginterpolator. When Euclidean distances increase, the respectiveweights decrease, thus limiting the contribution of the respectivecandidate neighboring pair in the final motion vector estimate.

B. Boundary Matching Error Minimization

In a second approach, the boundary matching criterion hasbeen employed in order to locate that interpolation scheme outof the four above-mentioned ones that estimates the “optimal”


motion vector with respect to the minimum boundary matchingerror [13] and thus leads to the best possible concealment, asshown in (13), at the bottom of the page, where denotesthe block size, the spatial coordinates of the top-leftpixel of the lost block, the reference frame (forward or back-ward), the current frame, and ,the estimated motion vector by each one of the four interpola-tion schemes.

C. Interpolation in the Direction of Minimum Change

In a third approach, interpolation has been attempted in thedirection of minimal change while preserving the transition inthe direction of maximal change inside a predefined motionfield neighborhood, in an inverse manner than that of method[19], which aims at image enhancement using rational controlfunctions of the rate of change of the local image content.In order to find the rate and direction of change (maximal orminimal) inside a predefined motion vector neighborhood, anapproach similar to the one proposed in [20] has been adopted.Let , be a two-valued two-dimensionalfunction denoting the estimated motion field of a frame, i.e.,

, where and ,, represent the functions of the horizontal and vertical

displacements composing the motion vectors at points .When the Euclidean distance of two points andtends to zero, the difference of the values of at thosepoints, , becomes the arcelement

(14)

Its squared norm, known as the first fundamental form, is givenby

(15)

where

(16)

According to [20], is a measure of the rate of mo-tion field change in a prespecified direction. The extremaof (15) are obtained in the direction of the eigenvectors of

the matrix and the values attained there are the

corresponding eigenvalues. Thus, the eigenvectors provide thedirection of maximal/minimal change at a given point (and , respectively), whereas the eigenvalues present themaximal/minimal rate of change ( and , respectively).The latter, as reported in [20], are estimated by

(17)

For the calculation of the maximal/minimal rates of change,(17) implies the prior estimation of the local gradients withinthe expressions of , where . For such purpose, di-rectional operators (referred to as convolution kernels as well)are defined for each one of six preselected directions within thelocal motion vector neighborhood: horizontal, vertical, diago-nals 45 , 135 , 30 , and 120 , in a manner similar to the onepresented in [21], which targets edge detection in color images.Such preselection is deemed necessary since no a priori knowl-edge exists on which direction the local motion field attains thebest smoothness properties. For each direction type, due to thevector-valued case, there is more than one rate of change sincethe minimal rate of change is not 0 [20]. The selection of the”edge-sensing” masks was done bearing in mind the particularsof the lost motion information problem (e.g., missing horizontalneighboring motion vectors). The masks are shown in Table I.The local motion field neighborhood employed for gradient es-timation (where the directional operators are applied) is of size3 3. The goal is to locate that direction toward which the localmotion field neighborhood presents the best smoothness prop-erties. Thus, for every gradient direction type, a minimum anda maximum rate of change, and , , areestimated by (17), based on the already undertaken approachesof [20], [21] for vector-valued cases. For single-valued cases,the minimum rate of change is 0 and only the maximum rateof change is applicable. The differences presentan appropriate criterion for detecting transitions in the local mo-tion field, as already stated in [20], since a large difference valuemeans that an irregular motion region exists in this position (thechange in motion is significant toward every direction), while asmall value means that we are in a smooth motion region (thechange in motion is similar toward all directions). Our aim is tolocate such smoothness properties, if existent within the localneighborhood, and perform interpolation toward that direction(the respective rational weight will be of large value in thiscase, as it will be later shown). For such purpose, such differ-ences have been chosen to control the estimation of the rationalweights in the motion vector rational interpolation approach.Specifically, candidate motion vector estimates are defined forevery direction type as the average of those neighboring motionvectors employed by the respective mask (nonzero elements).

(13)


TABLE IDIRECTIONAL OPERATORS USED TO ESTIMATE GRADIENT VALUES IN

DIFFERENT DIRECTIONS

For example, for type II, the candidate motion vector estimateis given by . In the case of motion

uniformity among neighbors, their average presents a good es-timate for the lost motion vector. The motion vector estimate inthis case is calculated by

(18)

where the rational weights are estimated by

(19)

D. Incorporation of Bilinear Interpolation Principles

Motivated by the approach of [12], the MVRI EC has beenextended to include bilinear interpolation principles in order tofurther exploit spatial correlations. The neighborhood employedin the motion vector rational-bilinear interpolation approach isshown in Fig. 3. The only difference with the neighborhood ofFig. 1 is that a finer interpolation grid is defined according towhich one motion vector per 4 4 block inside the lost 16 16MPEG-2 MB is estimated. Thus, 16 motion vectors , insteadof 1, are derived per lost MB. Consequently, the final motionestimate is a 4 4 matrix , . Theestimation scheme, a combination of vector rational and bilinearinterpolation, involves the computation of each

(20)

Fig. 3. Graphical presentation of the motion field estimation basis in theMVRI-BMFI method.

where . Equation (20) is an ex-tension of the vector rational interpolation case, presentedpreviously. are the rational interpolation weights eval-uated by (8), which are calculated only once per lost MB,since they are independent of spatial locations. representweights based on spatial locations. For each , a 2 3 matrix

, , is defined that contains thespatial location weights. The structure of is proportional tothe spatial localization of with respect to the motion vectorneighborhood in Fig. 3. Obviously, 16 such matrices, one per

, have to be calculated. They are defined as spatial locationlook-up tables, because they need to be estimated only once,due to the identical spatial structure of each inside the mo-tion vector neighborhood for dinstinct lost MBs. This results incomputational savings. The spatial weight calculation is donein an identical manner as in bilinear interpolation. Distancesfrom the estimated motion vector to its neighbor estimator aremeasured in the motion vector coordinate system (see Fig. 3).These are converted to integers for faster computations andless memory requirements. To help the reader understand theestimation process of , we present an example of howis estimated, as follows:

(21)When an array term is derived from the multiplication of twoindividual terms, the first one corresponds to the horizontal spa-tial distance from the existing neighboring motion vector sub-tracted from 1 and the second one to the vertical such distancesubtracted from 1.

This interpolation scheme attempts to estimate lost motioninformation in such a way that motion smoothness is attainedin smooth motion areas (linear operation performs best in sucha case), whereas irregular motion of adjacent blocks does notresult in high estimation errors (spatial correlations performwell in such cases). Thus, the method attains good motion fieldcharacteristics: motion smoothness and motion discontinuity, aslong as adjacent motion data is sufficient for correct estimation.Again, intra-coded neighbors are simply considered as havingzero valued motion vectors and no coding mode information ofadjacent blocks is actually exploited in the estimation process.


TABLE IICORRESPONDENCE OF ABBREVIATIONS AND INTRODUCED CONCEALMENT METHODS. MVRI ALWAYS STANDS FOR

MOTION VECTOR RATIONAL INTERPOLATION AND EC FOR ERROR CONCEALMENT

E. Exploitation of Available Coding Mode Information

This scheme additionally exploits available neighboringcoding mode information in order to consider solely theinter-coded neighbors in the motion vector rational interpola-tion approach. Thus, estimation errors due to the intra-coding ofneighboring blocks are avoided. The final estimate is evaluatedby

(22)

additionally constraining , to be the motion vectors of inter-coded neighbors. After the latter have been singled out basedon coding mode information, all possible pairs are con-structed and employed in (22). The additional use of codingmode information proves to enhance the estimation accuracy.

IV. SIMULATION RESULTS

In order to evaluate the performance of the motion vector ra-tional interpolation methods for EC, three different CCIR 601sequences at 4:2:0 chroma sampling format have been used,namely the Flower Garden (125 frames), the Mobile & Calendar(40 frames), and the Football (50 frames) sequences. These se-quences differ in the type of observed motion, as well as content.The first one is characterized by large uniform global motion(moving camera), the second one by medium-valued nonuni-form motion (three objects moving differently), while the lastone is characterized by highly irregular motion in many direc-tions. They have been coded at 5 Mbps at 25 fps (PAL) usingslice sizes equal to an entire row of MBs and setting the numberof frames in a group of pictures (GOP) equal to 12 and thenumber of frames between successive I- and P-frames or P- andP-frames equal to 3. Layered coding is not employed and mo-tion compensation is performed on a MB basis during encoding(one motion vector per 16 16 block for the luminance compo-nent). MPEG-2 transport packets are considered, which are 188bytes long composed by 4 bytes of header information and 184bytes of payload. A PER value of 2% has been considered. Er-rors may result from either isolated bit errors or packet loss. Theerror locations are assumed known and it is assumed that, oncean error is detected, all packets are discarded until the decoderis able to resynchronize. In order to help the reader quickly un-derstand the abbreviations used in the sequel for the methodsintroduced in this paper, Table II is presented.

TABLE IIIAVERAGE PSNR VALUES MEASURED ON THE THREE TEST SEQUENCES (Y

COMPONENT) AFTER THEIR CONCEALMENT BY THE METHODS UNDER

STUDY FOR A PER VALUE EQUAL TO 2%

Objective performance evaluation of the presented conceal-ment method is based on average PSNR values, whereas sub-jective evaluation is achieved by observing the visual quality ofthe concealed sequence. In order to assess the performance ofthe various motion field estimation processes, the Motion FieldEstimation Error [ ] is used

(23)

In (23), represents the total number of block motionvectors in a frame. represents the original codingmode of the considered MB, which should actually indicate toan inter-coded MB, i.e., forward predicted (F), backward pre-dicted (B), or bi-directionally predicted (D) (original motion in-formation available in this case). is the originalmotion field of the current frame , whereas isthe estimated one. Finally, since fast concealment is a require-ment for real-time decoding capabilities, the execution time foreach of the concealment methods under study is measured.

Table III illustrates the average PSNR values for the lumi-nance component evaluated on the concealed test sequences by


Fig. 4. Frame 25 of the Flower Garden sequence: (a) without errors, (b) witherrors,PER = 2%, concealed by: (c) MC-VM EC, (d) BMA EC, (e) combinedBMFI EC, and (f) MVRI-CodM EC.

using all the motion field estimation EC methods under study. Itcan be seen that, in almost all cases, the motion vector rationalinterpolation EC schemes as well as the F-B BM EC method at-tain the best result. The one that distinguishes itself, with respectto PSNR values, among the motion vector rational interpolationapproaches is the MVRI-CodM EC method. Its performance isattributed to the fact that this method additionally exploits avail-able neighboring coding mode information, which proves notto be redundant but rather of significance. Their rather satisfac-tory performance, compared with that of the other motion esti-mation EC methods under study, can be further established byobserving the achieved visual quality of the concealed test se-quences. In Figs. 4–6, certain frames of the concealed test se-quences Flower Garden, Mobile & Calendar, and Football, re-spectively, are shown. The methods used for concealment arethe MC-VM EC, the BMA EC, the Combined BMFI EC andthe MVRI-CodM EC, which lead to the best results with re-spect to PSNR values shown in Table III. The conclusion isthat the motion vector rational interpolation EC schemes leadto satisfactory concealment for cases of medium PER valuesand achieve smooth concealment, edge and line reconstruction,even in strong motion or irregular motion areas, provided thatthe neighboring motion information is adequate for the estima-tion of the lost data. In regions of strong or irregular motion,many of the other state-of-the-art temporal EC methods lead toobservable frame content shifts.

Table IV illustrates the temporal MFE average values evalu-ated on the estimated motion fields of predictively coded framesand only for the predictively coded MBs. It is easily observed

Fig. 5. Frame 26 of the Mobile & Calendar sequence: (a) without errors,(b) with errors, PER = 2%, concealed by: (c) MC-VM EC, (d) BMA EC,(e) combined BMFI EC, and (f) MVRI-CodM EC.

Fig. 6. Frame 25 of the Football sequence: (a) without errors, (b) with errors,PER = 2%, concealed by: (c) MC-VM EC, (d) BMA EC, (e) combined BMFIEC, and (f) MVRI-CodM EC.

that the MVRI EC methods, especially the one exploiting avail-able coding mode information, lead to small error values, a fact


TABLE IVAVERAGE MFE VALUES MEASURED ON THE ESTIMATED MOTION FIELDS OF

THE PREDICTIVELY CODED FRAMES OF THE THREE TEST SEQUENCES

Fig. 7. Backward motion field of frame 8 (B-frame) of the Flower Gardensequence: (a) without errors, (b) with errors, PER = 2%, estimated by: (c)MC-VM EC and (d) MVRI-CodM EC.

that justifies their remarkable property of being able to nonlin-early adapt their behavior with respect to local motion informa-tion and further establishes the observation that coding modeinformation is rather significant for EC purposes. In order tovisually comprehend the performance of the motion field esti-mation methods, Fig. 7 shows the original, the erroneous, andthe estimated motion fields produced by the MC-VM EC andMVRI-CodM EC methods. The erroneous or estimated part isincluded in a drawn rectangular with dashed lines. The observa-tion of the estimated motion fields leads to the conclusion thatthe MVRI EC methods manage to retain the two characteris-tics of motion fields: motion uniformity in uniformly movingregions and motion discontinuity in irregularly moving regions,provided that the available neighboring motion information isadequate for estimating lost motion information.

Since real-time implementation capabilities are of great im-portance in order to allow decoding and concealment in real-time and in parallel, e.g., in digital TV signal transmission andreception, the processing time requirements of the presented EC

TABLE VEXECUTION TIMES IN SECONDS, REQUIRED FOR MOTION FIELD ESTIMATION

BY THE EC METHODS UNDER STUDY, FOR ENTIRE SEQUENCE CONCEALMENT

methods are also examined. Execution time evaluation has beencarried on a SGI Workstation R4400 at 200 MHz with 64-MBRAM. The results are tabulated in Table V. It is seen that theMVRI EC methods are rather fast, combining good performanceand fast implementation, compared with other motion estima-tion EC methods under study. The MVE-BO EC and F-B BMEC methods present the slowest solutions due to the search theyperform to re-estimate lost motion information.

Finally, performance comparison among the different pre-sented motion vector rational interpolation EC schemes followsaiming at clarifying which may actually be used under certainconditions. By observing all tables with PSNR, MFE, and ex-ecution time values and having visually compared many con-cealed frames by each one of the presented methods, we are ledto the following remarks.

• The MVRI method that further exploits coding mode in-formation achieves the best results with respect to PSNRand MFE values with only a small increase in the requiredcomputational time which still is low compared to the onerequired by state-of-the-art concealment methods.

• The second best proves to be the one interpolating in thedirection of minimum change along with the one furtherusing bilinear interpolation aspects, the latter being onlyslightly slower than the former.

• The MVRI method that also uses coding mode informa-tion leads to enhanced visual quality of the concealedframes.

• The above performances are almost similarly monitoredfor all test sequences considered.

Based on the above remarks, one could concluded that the bestoption among the MVRI methods to be used is the one ex-ploiting coding mode information. However, when time con-straints are rather demanding and since the performance of theother MVRI methods is not far behind, faster MVRI methodscould be used.

V. CONCLUSIONS

Motion field estimation by motion vector rational interpola-tion has been investigated for EC purposes. A number of suchinterpolation schemes have been introduced. Motion vector


rational interpolation achieves to capture neighboring motionsmoothness or irregularities and adapt its behavior with respectto the available information. It is proven that all MVRI schemesperform satisfactorily and are remarkably fast compared withother motion field estimation EC methods under study. TheMVRI EC scheme that has the best interpolation performanceadditionally uses available adjacent coding mode information.The presented methods are able to deal with cases of low tomedium PER values.

Another possible application of motion vector rational in-terpolation, apart from EC, could be the smoothing of motionfields, evaluated, e.g., by block matching approaches which areknown to lead to many estimation errors, and their improvedconversion from coarser to finer ones.

REFERENCES

[1] Y. Wang and Q.-F. Zhu, “Error control and concealment for video com-munication: A review,” Proc. IEEE, vol. 86, pp. 974–997, May 1998.

[2] P. Salama, N. B. Shroff, and E. J. Delp, “Error concealment in encodedvideo streams,” in Signal Recovery Techniques for Image and VideoCompression and Transmission. Norwell, MA: Kluwer, 1998, pp.199–233.

[3] X. Lee, Y. Zhang, and A. Leon-Garcia, “Information loss recovery forblock-based image coding techniques – A fuzzy logic approach,” IEEETrans. Image Processing, vol. 4, pp. 259–273, Mar. 1995.

[4] H. Sun and W. Kwok, “Concealment of damaged block transform codedimages using projections onto convex sets,” IEEE Trans. Image Pro-cessing, vol. 4, pp. 470–477, Apr. 1995.

[5] W.-M. Lam and A. R. Reibman, “An error concealment algorithm forimages subject to channel errors,” IEEE Trans. Image Processing, vol.4, pp. 533–542, May 1995.

[6] J. W. Park, J. W. Kim, and S. Lee, “DCT coefficients recovery-basederror concealment technique and its application to the MPEG-2 bitstream error,” IEEE Trans. Circuits Syst. Video Technol., vol. 7, pp.845–854, Dec. 1997.

[7] H. Sun, J. W. Zdepski, W. Kwok, and D. Raychaudhuri, “Error con-cealment algorithms for robust decoding of MPEG compressed video,”Signal Processing: Image Commun., vol. 10, no. 4, pp. 249–268, 1997.

[8] G.-S. Yu, M. M.-K. Liu, and M. W. Marcellin, “POCS-based error con-cealment for packet video using multiframe overlap information,” IEEETrans. Circuits Syst. Video Technol., vol. 8, pp. 422–434, Aug. 1998.

[9] Z. Wang, Y. Yu, and D. Zhang, “Best neighborhood matching: An in-formation loss restoration technique for block-based image coding sys-tems,” IEEE Trans. Image Processing, vol. 7, pp. 1056–1061, July 1998.

[10] S. Tsekeridou and I. Pitas, “MPEG-2 error concealment based on blockmatching principles,” IEEE Trans. Circuits Syst. Video Technol., vol. 10,pp. 646–658, June 2000.

[11] J. Zhang, J. F. Arnold, and M. R. Frater, “A cell-loss concealmenttechnique for MPEG-2 coded video,” IEEE Trans. Circuits Syst. VideoTechnol., vol. 10, pp. 659–665, June 2000.

[12] M. Al-Mualla, N. Canagarajah, and D. R. Bull, “Temporal error con-cealment using motion field interpolation,” Electron. Lett., vol. 35, no.3, pp. 215–217, Feb. 1999.

[13] W.-M. Lam, A. R. Reibman, and B. Liu, “Recovery of lost or erroneouslyreceived motion vectors,” in Proc. 1993 Int. Conf. Acoustics, Speech andSignal Processing, vol. V, Apr. 1993, pp. 417–420.

[14] L. Khriji, F. A. Cheikh, and M. Gabbouj, “High-resolution digital re-sampling using vector rational filters,” SPIE Opt. Eng., vol. 38, no. 5,pp. 893–901, May 1999.

[15] F. A. Cheikh, L. Khriji, M. Gabbouj, and G. Ramponi, “Color imageinterpolation using vector rational filters,” in Proc. SPIE/EI Conf., Non-linear Image Processing IX, vol. 3304, San Jose, CA, Jan. 1998.

[16] L. Khriji, F. A. Cheikh, and M. Gabbouj, “Multistage vector rationalinterpolation for color images,” in Proc. 2nd IMACS-IEEE Int. Mul-ticonf. Computational Engineering in System Application (CESA’98),Hammamet, Tunisia, Apr. 1998.

[17] I.-W. Tsai and C.-L. Huang, “Hybrid cell loss concealment methods forMPEG-II-based packet video,” Signal Processing: Image Commun., vol.9, no. 2, pp. 99–124, 1997.

[18] S. Tsekeridou, F. A. Cheikh, M. Gabbouj, and I. Pitas, “Motion field esti-mation by vector rational interpolation for error concealment purposes,”in Proc. 1999 IEEE Int. Conf. Acoustics, Speech and Signal Processing,Phoenix, AZ, Mar. 1999.

[19] F. A. Cheikh, L. Khriji, and M. Gabbouj, “Unsharp masking-based ap-proach for color image processing,” in Proc. IX European Signal Pro-cessing Conf. (EUSIPCO’98), vol. II, Rhodes, Greece, Sept. 1998, pp.1033–1036.

[20] G. Sapiro and D. L. Ringach, “Anisotropic diffusion of multivalued im-ages with applications to color filtering,” IEEE Trans. Image Processing,vol. 5, pp. 1582–1586, Nov. 1996.

[21] J. Scharcanski and A. N. Venetsanopoulos, “Edge detection of colorimages using directional operators,” IEEE Trans. Circuits Syst. VideoTechnol., vol. 7, pp. 397–401, Apr. 1997.

Sofia Tsekeridou (M’04) received the Dipl. Elect. Eng. in 1996 and the Ph.D.degree in informatics in 2001, both from Aristotle University of Thessaloniki,Thessaloniki, Greece.

During 1996–2001, she was involved in research activities under Europeanand Greek funded R&D Projects, served as a Teaching Assistant and was alsoa Visiting Researcher at the Tampere University of Technology, Tampere, Fin-land. During 2001–2003, she was employed at the Development Programs De-partment, INTRACOM S.A., Greece, involved in R&D Projects dealing withnext generation enhanced and interactive digital TV. Since 2003, she holds theposition of Lecturer at the Electrical and Computer Engineering Department,Democritus University of Thrace, Xanthi, Greece. Her research interests lie inthe areas of signal, image and video processing, computer vision, multimediainformation systems, content- and semantic-based multimedia analysis and de-scription. She co-authored over 40 publications, has contributed to the TV Any-time standardization body, and serves as a reviewer for many international con-ferences and scientific journals.

Dr. Tsekeridou is a member of the Technical Chamber of Greece.

Faouzi Alaya Cheikh received the B.S. degree in electrical engineering in 1992from Ecole Nationale d’Ingenieurs de Tunis, Tunis, Tunisia, and the M.S. andthe Ph.D. degrees in electrical engineering from Tampere University of Tech-nology, Tampere, Finland in 1996 and 2004, respectively.

He is currently a Senior Researcher at the Institute of Signal Processing, Tam-pere University of Technology, Tampere, Finland. From 1994 to 1996, he wasa Research Assistant at the Institute of Signal Processing, and since 1997, hasbeen a Researcher with the same institute. His research interests include non-linear signal and image processing and analysis, pattern recognition and con-tent-based analysis and retrieval. He has been an active member in many Finnishand European research projects among them Nobless esprit, COST 211 quat,and MUVI. He co-authored over 30 publications.

Dr. Cheikh served as Associate Editor of the EURASIP Journal on AppliedSignal Processing for the Special Issue on Image Analysis for Multimedia In-teractive Services. He serves as a reviewer for several conferences and journals.

Moncef Gabbouj (SM’95) received the B.S. degree in electrical engineeringfrom Oklahoma State University, Stillwater, in 1985, and the M.S. and Ph.D.degrees in electrical engineering from Purdue University, West Lafayette, IN,in 1986 and 1989, respectively.

He is currently a Professor and Head of the Institute of Signal Processingof Tampere University of Technology, Tampere, Finland. His research interestsinclude nonlinear signal and image processing and analysis, content-based anal-ysis and retrieval and video coding. He is co-author of over 250 publications.

Dr. Gabbouj served as Associate Editor of the IEEE TRANSACTIONS ON

IMAGE PROCESSING. He is the past chairman of the IEEE Finland Section andthe IEEE Circuits and Systems (CAS) Society, TC DSP, and the IEEE SignalProcessing/CAS Finland Chapter. He was co-recipient of the Myril B. ReedBest Paper Award from the 32nd Midwest Symposium on Circuits and Systemsand co-recipient of the NORSIG 94 Best Paper Award from the 1994 NordicSignal Processing Symposium.


Ioannis Pitas (SM’94) received the Dipl. Elect. Eng. in 1980 and the Ph.D. de-gree in electrical engineering in 1985, both from the University of Thessaloniki,Thessaloniki, Greece.

Since 1994, he has been a Professor at the Department of Informatics, Uni-versity of Thessaloniki, Greece. His current interests are in the areas of digitalimage processing, multimedia signal processing, multidimensional signal pro-cessing and computer vision. He has published over 450 papers, contributed in17 books and authored, co-authored, edited, co-edited 7 books in his area ofinterest. HE is the co-author of the books Nonlinear Digital Filters: Principlesand Applications (Norwell, MA: Kluwer, 1990) and 3D Image Processing Al-gorithms (New York: Wiley, 2000), is the author of the books Digital ImageProcessing Algorithms (Englewood Cliffs, NJ: Prentice Hall, 1993), DigitalImage Processing Algorithms and Applications (New York: Wiley, 2000), Dig-ital Image Processing (in Greek, 1999), and is the editor of the book ParallelAlgorithms and Architectures for Digital Image Processing, Computer Visionand Neural Networks (New York: Wiley, 1993) and co-editor of the book Non-linear Model-Based Image/Video Processing and Analysis (New York: Wiley,2000). He is/was principal investigator/researcher in more than 40 competitiveR&D projects and in 11 educational projects, all mostly funded by the EuropeanUnion.

Dr. Pitas is/was Associate Editor of the IEEE TRANSACTIONS ON

CIRCUITS AND SYSTEMS, IEEE TRANSACTIONS ON NEURAL NETWORKS,IEEE TRANSACTIONS ON IMAGE PROCESSING, IEICE, Circuits Systems andSignal Processing, and was co-editor of Multidimensional Systems and SignalProcessing. He was Chair of the 1995 IEEE Workshop on Nonlinear Signaland Image Processing (NSIP95), Technical Chair of the 1998 European SignalProcessing Conference, and General Chair of IEEE ICIP2001. He was co-chairof the 2003 International workshop on Rich media content production. He wastechnical co-chair of the 2003 Greek Informatics conference (EPY).

Documents

876 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 6, …moncef/publications/vector-rational-interpolation.pdf · supported by the European RTN Project RTN1–1999-00177 ... of color image